Data-Driven Sparse Sensor Placement for Reconstruction

Krithika Manohar∗, Bingni W. Brunton, J. Nathan Kutz, and Steven L. Brunton∗Corresponding author: kmanohar@uw.edu

Abstract

Optimal sensor placement is a central challenge in the design, prediction, estimation, and control ofhigh-dimensional systems. High-dimensional states can often leverage a latent low-dimensional represen-tation, and this inherent compressibility enables sparse sensing. This article explores optimized sensorplacement for signal reconstruction based on a tailored library of features extracted from training data.Sparse point sensors are discovered using the singular value decomposition and QR pivoting, which aretwo ubiquitous matrix computations that underpin modern linear dimensionality reduction. Sparse sensingin a tailored basis is contrasted with compressed sensing, a universal signal recovery method in which anunknown signal is reconstructed via a sparse representation in a universal basis. Although compressedsensing can recover a wider class of signals, we demonstrate the benefits of exploiting known patternsin data with optimized sensing. In particular, drastic reductions in the required number of sensors andimproved reconstruction are observed in examples ranging from facial images to fluid vorticity fields.Principled sensor placement may be critically enabling when sensors are costly and provides faster stateestimation for low-latency, high-bandwidth control.

Scalars NOTATION

n State dimensionm Number of snapshotsp Number of sensors (measurements)r Intrinsic rank of tailored basis Ψr

K Sparsity of state in universal basis Ψη Variance of zero-mean sensor noise

Vectorsx ∈ Rn High-dimensional statey ∈ Rp Measurements of statea ∈ Rr Tailored basis coefficientsej ∈ Rn Canonical basis vectors for Rn

s ∈ Rn K-sparse basis coefficientsγ ∈ Np Sensor placement indicesψ ∈ Rn POD modes (columns of Ψr)θ ∈ R1×r Rows of Θ

MatricesC ∈ Rp×n Measurement matrixQ Unitary QR factor matrixR Upper triangular QR factor matrixΨ ∈ Rn×n Universal basisΨr ∈ Rn×r Tailored basis of rank r

Θ = CΨ Product of measurement and basisX ∈ Rn×m Data matrix with m snapshots

Optimal sensor and actuator placement is animportant unsolved problem in control theory.Nearly every downstream control decision is af-fected by these sensor/actuator locations, but de-termining optimal locations amounts to an in-tractable brute force search among the combinato-rial possibilities. Indeed, there are

(np

)= n!

(n−p)!p!possible choices of p point sensors out of an n

dimensional state x. Determining optimal sensorand actuator placement in general, even for linearfeedback control, is an open challenge. Instead,sensor and actuator locations are routinely chosenaccording to heuristics and intuition. For moderatesized search spaces, the sensor placement prob-lem has well-known model-based solutions usingoptimal experiment design [1], [2], informationtheoretic and Bayesian criteria [3], [4], [5], [6],[7] . We explore how to design optimal sensorlocations for signal reconstruction in a frameworkthat scales to arbitrarily large problems, leverag-ing modern techniques in machine learning andsparse sampling. Reducing the number of sen-sors through principled selection may be criticallyenabling when sensors are costly, and may alsoenable faster state estimation for low latency, high

1

arX

iv:1

701.

0756

9v2

[m

ath.

OC

] 1

8 A

ug 2

017

2

Mathematical formulation of sensor selection

Many physical systems are described by a high-dimensional state x ∈ Rn, yet the dynamics evolveon a low-dimensional attractor that can be leveraged for prediction and control. Thus, a state x thatevolves according to nonlinear dynamics x(t) = f(x(t)) will often have a compact representation ina transform basis Ψ. In a universal basis Ψ ∈ Rn×n, such as Fourier or wavelet bases, x may havea sparse representation

x = Ψs s ∈ Rn, (S1)

where s is a sparse vector indicating which few modes of Ψ are active. In a tailored basis Ψr ∈ Rn×r,such as a proper orthogonal decomposition (POD) basis, x may have a low-rank representation

x = Ψra a ∈ Rr. (S2)

The central challenge in this work is to design a measurement matrix C ∈ Rp×n consisting of a smallnumber (p� n) of optimized measurements

y = Cx y ∈ Rp, (S3)

that facilitate accurate reconstruction of either s or a, and hence x. Combining (S1) and (S3) yields

y = (CΨ)s = Θs, (S4)

which is referred to as the compressed sensing problem, while combining (S2) and (S3) yields

y = (CΨr)a = Θa. (S5)

In both cases, effective measurements C given a basis Ψ or Ψr are chosen so that the operator Θis well-conditioned for signal reconstruction. Thus, it is possible to solve for the sparse coefficientss or the low-rank coefficients a given the measurements y, either by `1 minimization in (S4) orpseudoinverse of Θ in (S5), respectively. The goal of this work is to optimize the measurementsin C. Moreover, in many physical applications, it is desired that C consists of rows of the identifymatrix, corresponding to individual point sensors of individual components of x.

bandwidth control.This article explores optimized sensor place-

ment for signal reconstruction based on a tai-lored library of features extracted from trainingdata. In this paradigm, optimized sparse sensorsare computed using a powerful sampling schemebased on the matrix QR factorization and singularvalue decomposition. Both procedures are nativelyimplemented in modern scientific computing soft-ware, and Matlab code supplements are providedfor all examples in this paper [8]. These data-driven computations are more efficient and easierto implement than the convex optimization meth-ods used for sensor placement in classical designof experiments. In addition, data-driven sensingin a tailored basis is contrasted with compressedsensing, a universal signal recovery method in

which an unknown signal is reconstructed usinga sparse representation in a universal basis. Al-though compressed sensing can recover a widerclass of signals, we demonstrate the benefits ofexploiting known patterns in data with optimizedsensing. In particular, drastic reductions in therequired number of sensors and improved recon-struction are observed in examples ranging fromfacial images to fluid vorticity fields. The overar-ching signal reconstruction problem is formulatedin “Sidebar: Mathematical formulation of sensorselection".

There are myriad complex systems that wouldbenefit from principled, scaleable sensor and ac-tuator placement, including fluid flow control [9],power grid optimization [10], epidemiologicalmodeling and suppression [11], bio-regulatory

3

network monitoring and control [12], and high-performance computing [13], [14], to name onlya few. In applications where individual sensorsare expensive, reducing the number of sensorsthrough principled design may be critically en-abling. In applications where fast decisions are re-quired, such as in high performance computing orfeedback control, computations may be acceleratedby minimizing the number of sensors required. Inother words, low-dimensional computations maybe performed directly in the sensor space.

Scaleable optimization of sensor and actuatorplacement is a grand challenge problem, withtremendous potential impact and considerablemathematical depth. With existing mathematicalmachinery, optimal placement can only be deter-mined in general using a brute-force combinatorialsearch. Although this approach has been success-ful in small-scale problems [15], a combinatorialsearch does not scale well to larger problems.Moore’s law of exponentially increasing computerpower cannot keep pace with this combinatorialgrowth in complexity.

Despite the challenges of sensing and actu-ation in a high-dimensional, possibly nonlineardynamical system, there are promising indicatorsthat this problem may be tractable with moderntechniques. High-dimensional systems, such as arefound in fluids, epidemiology, neuroscience, andthe power grid, typically exhibit dominant coher-ent structures that evolve on a low-dimensional at-tractor. Indeed, much of the success of modern ma-chine learning rests on the ability to identify andtake advantage of patterns and features in high-dimensional data. These low-dimensional patternsare often identified using dimensionality reduc-tion techniques [16] such as the proper orthogonaldecomposition (POD) [17], [18], [19], which is avariant of principal component analysis (PCA), ormore recently via dynamic mode decomposition(DMD) [20], [21], [22], [23], diffusion maps [24],[25], [26], etc. In control theory, balanced trunca-tion [27], balanced proper orthogonal decomposi-

tion (BPOD) [28], [29], and the eigensystem realiza-tion algorithm (ERA) [30], have been successfullyapplied to obtain control-oriented reduced-ordermodels for many high-dimensional systems.

In addition to advances in dimensionality re-duction, key developments in optimization, com-pression, and the geometry of sparse vectors inhigh-dimensional spaces are providing powerfulnew techniques to obtain approximate solutionsto NP-hard, combinatorially difficult problems inscaleable convex optimization architectures. Forexample, compressed sensing [31], [32], [33] pro-vides convex algorithms to solve the combina-torial sparse signal reconstruction problem withhigh probability. Ideas from compressed sensinghave been used to determine the optimal sen-sor locations for categorical decisions based onhigh-dimensional data [34]. Recently, compressedsensing, sparsity-promoting algorithms such as thelasso regression [35], [36], [37], and machine learn-ing have been increasingly applied to characterizeand control dynamical systems [38], [39], [40], [41],[42], [43], [44], [45], [46], [47]. These techniqueshave been effective in modeling high-dimensionalfluid systems using POD [48] and DMD [49], [50],[51], [52]. Information criteria [53], [54] has alsobeen leveraged for the sparse identification of non-linear dynamics [42], as in [55], [56], and may alsobe useful for sensor placement.

Thus, key advances in two fields are funda-mentally changing our approach to the acquisi-tion and analysis of data from complex dynamicalsystems: 1) machine learning, which exploits pat-terns in data for low-dimensional representations,and 2) sparse sampling, where a full signal canbe reconstructed from a small subset of measure-ments. The combination of machine learning andsparse sampling is synergistic, in that underlyinglow-rank representations facilitate sparse measure-ments. Exploiting coherent structures underlyinga large state space allows us to estimate and con-trol systems with few measurements and sparseactuation. Low-dimensional, data-driven sensing

4

and control is inspired in part by the high perfor-mance exhibited by biological organisms, such asan insect that performs robust, high-performanceflight control in a turbulent fluid environment withminimal sensing and low-latency control [57]. Theyprovide proof-by-existence that it is possible toassimilate sparse measurements and perform low-dimensional computations to interact with coher-ent structures in a high-dimensional system (i.e., aturbulent fluid).

Here we explore two competing perspectiveson high-dimensional signal reconstruction: 1) theuse of compressed sensing based on random mea-surements in a universal encoding basis, and 2) theuse of highly specialized sensors for reconstruc-tion in a tailored basis, such as POD or DMD.These choices are also discussed in the contextof feedback control. Many competing factors im-pact control design, and a chief consideration isthe latency in making a control decision, withlarge latency imposing limitations on robust per-formance [58], [59]. Thus, for systems with fastdynamics and complex coherent structures, it isimportant to make control decisions quickly basedon efficient low-order models, with sensors andactuators placed strategically to gather informationand exploit sensitivities in the dynamics.

Extensions to dynamics, control, and multiscalephysics

Data-driven sensor selection is generally usedfor instantaneous full-state reconstruction, despitethe fact that many signals are generated by adynamical system [60], [19]. Even in reduced-ordermodels, sensors are typically used to estimatenonlinear terms instantaneously without takingadvantage of the underlying dynamics. However,it is well known that for linear control systems [59],[58], the high-dimensional state may be recon-structed with few sensors, if not a single sensor,by leveraging the time history in conjunction witha model of the dynamics, as exemplified by theKalman filter [61], [62]. In dynamic estimation andcontrol, prior placement of sensors and actuators

is generally assumed. Extending the sensor place-ment optimization to the model reduction [27],[28], [29] and system identification [30], [63], [64],[65] of linear control systems is an important av-enue of ongoing work. In particular, sensors andactuators may be chosen to increase the volumeof the controllability and observability Gramians,related to the original balanced truncation liter-ature [27]. More generally, sensor and actuatorplacement may be optimized for robustness [66],[67], or for network control and consensus prob-lems [68], [69], [70], [71], [72].

The sensor placement algorithms discussedabove are rooted firmly in linear algebra, makingthem readily extensible to linear control systems.Recent advances in dynamical systems are provid-ing techniques to embed nonlinear systems in alinear framework through a suitable choice of mea-surement functions of the state, opening up the pos-sibility of optimized sensing for nonlinear systems.As early as the 1930s, Koopman demonstrated thata nonlinear system can be rewritten as an infinite-dimensional linear operator on the Hilbert space ofmeasurement functions [73]. This perspective didnot gain traction until modern computation anddata collection capabilities enabled the analysis oflarge volumes of measurement data.

Modern Koopman theory may drive sensorplacement and the selection of nonlinear measure-ment functions on the sensors to embed nonlin-ear dynamics in a linear framework for optimalnonlinear estimation and control. This approach isconsistent with neural control systems, where bio-logical sensor networks (e.g., strain sensors on aninsect wing) are processed through nonlinear neu-ral filters before being used for feedback control.Much of the modern Koopman operator theory hasbeen recently developed [74], [75], [76], [77], andit has been shown that under certain conditionsDMD approximates the Koopman operator [20],[21], [22], [23]; sensor fusion is also possible inthe Koopman framework [78]. Recently, Koopmananalysis has been used to develop nonlinear esti-

5

mators [79], [80] and controllers [81], although es-tablishing rigorous connections to control theory isan ongoing effort [82], [83], [84]. Koopman theoryhas also been used to analyze chaotic dynamicalsystems from time-series data [85], [86], relyingon the Takens embedding [87], which is related tosensor selection.

Beyond extending sensor selection to nonlin-ear systems and control, there is a significantopportunity to apply principled sensor selectionto multiscale systems. Turbulence is an impor-tant high-dimensional system that exhibits multi-scale phenomena [19], [88], [89]. Data-driven ap-proaches have been used to characterize turbulentsystems [9], including clustering [90], network the-ory [91], [92], DMD-based model reduction [93],[94], and local POD subspaces [95], to name afew. Recently, a multiresolution DMD has beenproposed [96], where a low-dimensional subspacemay locally characterize the attractor, despite ahigh-dimensional global attractor. This approachmay significantly reduce the number of sensorsneeded for multiscale problems.

Compressed sensing: Randommeasurements in a universal basisThe majority of natural signals, such as images

and audio, are highly compressible, meaning thatwhen the signal is written in an appropriate coor-dinate system, only a few basis modes are active.These few values corresponding to the large modeamplitudes must be stored for accurate reconstruc-tion, providing a significant reduction compared tothe original signal size. In other words, in the uni-versal transform basis, the signal may be approxi-mated by a sparse vector containing mostly zeros.This inherent sparsity of natural signals is cen-tral to the mathematical framework of compressedsensing. Signal compression in the Fourier domainis illustrated on an image example in “Sidebar:Image compression". Further, sparse signal recov-ery using compressed sensing is demonstratedon a sinusoidal example in “Compressed sensing

Sidebar: Image compression

Images and audio signals tend to be sparsein Fourier or wavelet bases, providing thefoundation of JPEG and MP3 compression,respectively. This is shown schematically inFig. F1 using the included Matlab code.

X = imread(’cappuccino’,’jpeg’);[nx,ny] = size(X);S = fft2(X); % X = Psi*sSsort = sort(abs(S(:)));thresh = Ssort(ceil(.95*nx*ny));bigind = abs(S)>thresh; % big coeffsStrunc = S.*bigind; % keep big coeffsXrecon = uint8(ifft2(Strunc));

F

F−1

Keep 5 %

Figure F1: Fourier image compression.

example". The theory of compressed sensing [32],[31], [97], [98], [99], [100], [33], [101] inverts thiscompression paradigm. Instead of collecting high-dimensional measurements just to compress anddiscard most of the information, it may be pos-sible to collect a low-dimensional subsample orcompression of the data and then infer the sparsevector of coefficients in the transformed coordinatesystem.

Theory of compressed sensingMathematically, a compressible signal x ∈ Rn

may be written as a sparse vector s ∈ Rn in a newbasis Ψ ∈ Rn×n such that

x = Ψs. (1)

The vector s is called K-sparse if there are exactlyK nonzero elements. To be able to represent any

6

= =

y C Ψ s Θ s

Figure 1: Compressed sensing provides the sparsest solution to an underdetermined linear system.

natural signal, rather than just those from a tai-lored category, the basis Ψ must be complete.

Consider a set of measurements y ∈ Rp, ob-tained via a measurement matrix C ∈ Rp×n, whichsatisfies

y = Cx = CΨs = Θs. (2)

In general, for p < n (2) is underdetermined, andthere are infinitely many solutions. The least leastsquaressquares (minimum ‖s‖2) solution is notsparse, and typically yields poor reconstruction.Instead, knowing that natural signals are sparse,we seek the sparsest s consistent with the mea-surements y,

s = argmins′

‖s′‖0, such that y = CΨs′, (3)

where ‖s‖0 is the `0 pseudo-norm correspondingto the number of non-zero entries of s. Unfor-tunately, this optimization problem is intractable,requiring a combinatorial brute-force search acrossall sparse vectors s. A major innovation of com-pressed sensing is a set of conditions on the mea-surement matrix C that allow the nonconvex `0-minimization in (3) to be relaxed to the convex `1-minimization [98], [102]

s = argmins′

‖s′‖1, such that y = CΨs′, (4)

where ‖s‖1 =∑nk=1 |sk|. This formulation is shown

schematically in Fig. 1.For the `1-minimization in (4) to yield the

sparsest solution in (3) with high probability, the

measurements C must be chosen so that Θ = CΨsatisfies a restricted isometry property (RIP)

(1− δK)‖s‖22 ≤ ‖CΨs‖2

2 ≤ (1 + δK)‖s‖22, (5)

where δK is a small positive restricted isometryconstant [103], [100]. In particular, there are twoconditions on C for a RIP to be satisfied for allK-sparse vectors s:

1) The measurements C must be incoherent withrespect to the basis Ψ. This incoherencemeans that the rows of C are sufficientlyuncorrelated with the columns of Ψ, as quan-tified by µ

µ(C,Ψ) =√nmax

j,k|〈ck,ψj〉|. (6)

Small µ indicates better incoherent measure-ments, with an optimal value of µ = 1. Here,ck denotes the k-th row of C and ψj the j-th column of Ψ, both of which are assumedto be normalized. A more detailed discussionabout incoherence and the RIP may be foundin [33], [100].

2) The number of measurements p must sat-isfy [99], [98], [33], [100], [104]

p ∼ O(K log(n/K)). (7)

The K log(n/K) term above is generally mul-tiplied by a small constant multiple of theincoherence. Thus, fewer measurements arerequired if they are less coherent.

7

Intuitively, the existence of a RIP impliesthat the geometry of sparse vectors is preservedthrough the measurement matrix CΨ. Determin-ing the exact constant δK may be extremely chal-lenging in practice, and it tends to be more de-sirable to characterize the statistical properties ofδK , as the measurement matrix C may be ran-domly chosen. “Sidebar: QR pivoting fails to ex-ploit sparsity" describes why it is not possible touse QR pivot locations as optimized sensors forcompressed sensing, since they fail to identify thesparse structure of an unknown signal.

Often, a generic basis such as Fourier orwavelets may be used to represent the signalsparsely. Spatially localized measurements (i.e.,single pixels in the case of an image) are opti-mally incoherent with respect to the Fourier basis,so that µ(C,Ψ) = 1. Thus, single pixel measure-ments are ideal because they excite a broadbandfrequency response. In contrast, a measurementcorresponding to a fixed Fourier mode would beuninformative; if the signal is not sparse in thisparticular frequency, this measurement providesno information about the other Fourier modes. Formany engineering applications, spatially localizedmeasurements are desirable, as they correspond tophysically realizable sensors, such as buoys in theocean.

One of the major results of compressed sens-ing is that random projection measurements ofthe state (i.e., entries of C that are Bernoulli orGaussian random variables) are incoherent withrespect to nearly any generic basis Ψ [99], [97],[32]. This result is truly remarkable; however, theincoherence of random projections is not optimal,and typically scales as µ ∼

√2 log(n). Moreover,

it may be difficult to obtain random projections ofthe full state x in physical applications.

There are many alternative strategies to solvefor the sparsest solution to (2). Greedy algorithmsare often used [105], [106], [107], [108], [109], in-cluding the compressed sampling matching pur-suit (CoSaMP) algorithm [110]. In addition, there

is additional theory about how sparse the randomprojections may be for compressed sensing [111],[112].

Compressed sensing exampleAs a simple example, we consider a sparse

signal that is constructed as the sum of threedistinct cosine waves,

x(t) = cos(2π×37t)+cos(2π×420t)+cos(2π×711t).(8)

The Shannon-Nyquist sampling theorem [113],[114] states that for full signal reconstruction,we must sample at twice the highest frequencypresent, indicating a theoretical minimum sam-pling rate of 1422 Hz. However, since the signalis sparse, we may sample at considerably lowerthan the Nyquist rate, in this case at an average of256 Hz, shown in Figure 2. Note that for accuratesparse signal reconstruction, these measurementsmust be randomly spaced in time, so that therelative spacing of consecutive points may be quiteclose or quite far apart. Spacing points evenly witha sampling rate of 256 Hz would alias the signal,resulting in poor reconstruction. Matlab code forreproducing Figure 2 is provided below.

N = 4096;

t = linspace(0, 1, N);

x = cos(2*pi*37*t) + cos(2*pi*420*t) +

cos(2*pi*711*t);

%% Randomly sample signal

p = 256; % num. samples, p=N/16

perm = randperm(N,p);

y = x(perm); % compressed measurement

%% Solve compressed sensing problem

Psi = dct(eye(N, N)); % build Psi

Theta = Psi(perm, :); % Measure Psi

cvx_begin; % L1-minimization with CVX

variable s(N);

minimize( norm(s,1) );

subject to

Theta*s == y’;

cvx_end;

xrecon = idct(s); % reconstruct x

8

0 200 400 600 800 10000

200

400

600

800

1000

1200

0.22 0.23 0.24 0.25 0.26-3

-2

-1

0

1

2

3

0.22 0.23 0.24 0.25 0.26-3

-2

-1

0

1

2

3

0 200 400 600 800 10000

200

400

600

800

1000

1200

0 200 400 600 800 10000

200

400

600

800

1000

1200

0.22 0.23 0.24 0.25 0.26-3

-2

-1

0

1

2

3

0.22 0.23 0.24 0.25 0.26-3

-2

-1

0

1

2

3

0 200 400 600 800 10000

200

400

600

800

1000

1200

0 200 400 600 800 10000

200

400

600

800

1000

1200

0.22 0.23 0.24 0.25 0.26-3

-2

-1

0

1

2

3

0.22 0.23 0.24 0.25 0.26-3

-2

-1

0

1

2

3

0 200 400 600 800 10000

200

400

600

800

1000

1200

0 200 400 600 800 10000

200

400

600

800

1000

1200

0.22 0.23 0.24 0.25 0.26-3

-2

-1

0

1

2

3

0.22 0.23 0.24 0.25 0.26-3

-2

-1

0

1

2

3

0 200 400 600 800 10000

200

400

600

800

1000

1200

Time [s] Frequency [Hz]

x

x

Power Spectral Density

Power Spectral Density

Figure 2: Compressed sensing, applied to three-tone signal.

Optimal sparse sensing in a tailoredbasis

The compressed sensing strategy above is idealfor the recovery of a high-dimensional signal ofunknown content using random measurements ina universal basis. However, if information is avail-able about the type of signal (e.g., the signal is aturbulent velocity field or an image of a humanface), it is possible to design optimized sensors thatare tailored for the particular signals of interest.Dominant features are extracted from a trainingdataset consisting of representative exaples, for ex-ample using the proper orthogonal decomposition(POD). These low-rank features, mined from pat-terns in the data, facilitate the design of specializedsensors that are tailored to a specific problem.

Low-rank embeddings, such as POD, havealready been used in the ROM community toselect measurements in the state space that areinformative for feature space reconstruction. Theso-called empirical interpolation methods seek thebest interpolation points for a given basis of PODfeatures. These methods have primarily been used

to speed up the evaluation of nonlinear terms inhigh-dimensional, parameterized systems. How-ever, the resulting interpolation points correspondto measurements in state space, and their usefor data-driven sensor selection has largely beenoverlooked. We will focus on this formulation ofsensor selection and explore sparse, convex, andgreedy optimization methods for solving it.

We begin with brief expositions on POD andour mathematical formulation of sensor place-ment, followed by an overview of related workin design of experiments and sparse sampling. Weconclude this section with our generalized sensorselection method that connects empirical interpo-lation methods, such as QR pivoting to optimizecondition number, with D-optimal experimentaldesign [117]. The QR pivoting method describedin “Sparse sensor placement with QR pivoting"is particularly favorable, as it is fast, simple toimplement, and provides nearly optimal sensorstailored to a data-driven POD basis. Finally, thedistinctions between compressed sensing and ourdata-driven sensing are summarized in “Sidebar:

9

Sidebar: Proper orthogonal decomposition and eigenfaces

One of the most visually striking and intuitive applications of proper orthogonal decomposition(POD) is the feature extraction of facial images. These POD eigenmodes of these datasets are calledeigenfaces due to their resemblance to generic human faces. We demonstrate this application of PODon the extended Yale B dataset [115], [116], consisting of cropped and aligned images of severalindividuals in different lighting conditions. We obtain a resized version of the dataset in the formof Matlab data files from [S1]. Each image is a 32 × 32 matrix of grayscale pixel values, reshapedinto a column vector of length 1024 and assembled into a data matrix X. This example, detailedin “Extended Yale B eigenfaces", is a benchmark problem for sensor selection.Matlab code for obtaining eigenfaces from training images is provided. First, training images areused to assemble a mean-subtracted data matrix.

meanface = mean(X,2);X = X-repmat(meanface,1,size(X,2)); % mean centered data

Next, POD eigenfaces are obtained using the singular value decomposition of the data matrix.Outputs from both code snippets are visualized in Figure F2.

% proper orthogonal decomposition[Psi,S,V] = svd(X,’econ’);eigenfaces = Psi(:,1:10);

Training images from the Extended Yale B dataset

First ten POD eigenfaces

Figure F2: Proper orthogonal decomposition (POD) modes successfully recover important facialinformation such as the main facial features (eyes, nose, mouth) followed by depth information(brows, ridges, chin).

References[S1] Available online (last accessed 7/1/2017)

http://www.cad.zju.edu.cn/home/dengcai/Data/FaceData.html

Comparison – Sparse sensing methods".

Proper orthogonal decompositionPOD is a widespread data-driven dimension-

ality reduction technique [18], [19] used in manydomains; it is also commonly known as theKarhunen-Loève expansion, principal componentanalysis (PCA) [118], and empirical orthogonalfunctions [119]. POD expresses high-dimensionalstates x ∈ Rn as linear combinations of a smallnumber of orthonormal eigenmodes ψ (i.e., PODmodes) that define a low-dimensional embeddingspace. States are projected into this POD subspace,yielding a reduced representation that can be used

to streamline tasks that would normally be ex-pensive in the high-dimensional state space. Thislow-rank embedding does not come for free, butinstead requires training data to tailor the PODbasis to a specific problem. POD is illustrated ona simple example of extracting coherent featuresin images of human faces in “Sidebar: Properorthogonal decomposition and eigenfaces".

A low-dimensional representation of x interms of POD coefficients a can be lifted back tothe full state with a linear combination of POD

10

modes,

xi ≈r∑

k=1ak(ti)ψk(x).

For time-series data xi, the coefficients ak(ti) varyin time and ψk(x) are purely spatial modes with-out time dependence, resulting in a space-timeseparation of variables. Thus, care should be takenapplying POD to data from a traveling wave prob-lem.

The eigenmodes ψk and POD coefficients akare easily obtained from the singular value de-composition (SVD). Given a data matrix of statespace observations X = [x1 x2 . . . xm], the result-ing eigenmodes are the orthonormal left singularvectors Ψ of X obtained via the SVD,

X = ΨΣVT ≈ ΨrΣrVTr . (9)

The matrices Ψr and Vr contain the first r columnsof Ψ and V (left and right singular vectors, respec-tively), and the diagonal matrix Σr contains thefirst r × r block of Σ (singular values). The SVDis the optimal least squares approximation to thedata for a given rank r, as demonstrated by theEckart-Young theorem [120]

X? = argminX

‖X− X‖F s. t. rank(X) = r, (10)

where X? = ΨrΣrVTr , and ‖ · ‖F is the Frobenius

norm. The low-dimensional vector of POD coef-ficients for a state x is given by the orthogonalprojection a = ΨT

r x. Thus, the POD is a widelyused dimensionality reduction technique for high-dimensional systems. This reduction allows com-putational speedup of numerical time-stepping,parameter estimation, and control.

Choosing the intrinsic target rank withoutmagnifying noise in the data is a difficult task.In practice, r is often chosen by thresholding thesingular values to capture some percentage of thevariance in the data. An optimal hard thresholdis derived in [121] based on the singular valuedistribution and aspect ratio of the data matrix, as-suming additive Gaussian white noise of unknown

variance. This threshold criterion has been effectivein practice, even in cases where the noise is likelynot Gaussian.

Sensor placement for reconstructionWe optimize sensor placement specifically to

reconstruct high-dimensional states from pointmeasurements, given data-driven or tailored bases.Recall that full states may be expressed as anunknown linear combination of basis vectors

xj =r∑

k=1Ψjkak, (11)

where Ψjk is the coordinate form of Ψr from (S2).Effective sensor placement results in a point mea-surement matrix C that is optimized to recoverthe modal mixture a from sensor outputs y. Pointmeasurements require that the sampling matrixC ∈ Rp×n be structured in the following way

C =[eγ1 eγ2 . . . eγp

]T, (12)

where ej are the canonical basis vectors for Rn

with a unit entry at index j and zeros elsewhere.Note that point measurements are fundamentallydifferent than the suggested random projectionsof compressive sensing. The measurement matrixresults in the linear system

yi =n∑j=1

Cijxj =n∑j=1

Cij

r∑k=1

Ψjkak, (13)

where Cij is the coordinate form of C from (S3).The observations in y consist of p elements selectedfrom x

y = Cx = [xγ1 xγ2 . . . xγp]T , (14)

where γ = {γ1, . . . , γp} ⊂ {1, . . . , n} denotes theindex set of sensor locations with cardinality |γ| =p.

When x is unknown, it can be reconstructedby approximating the unknown basis coefficientsa with the Moore-Penrose pseudoinverse, a =Θ†y = (CΨr)†y. Equivalently, the reconstruction

11

Sidebar: Condition number

The condition number of a matrix Θ gives a measure of how sensitive matrix multiplication orinversion is to errors in the input, with larger condition number indicating worse performance. Thecondition number κ(Θ) is the ratio of the maximum and minimum singular values of Θ

κ(Θ) = σmax(Θ)σmin(Θ) . (S6)

To see the effect of the condition number on matrix multiplication, consider a square, invertible Θand an input signal x that is contaminated by noise εx. Further, we consider the worst-case scenariowhere x is aligned with the right singular vector of Θ corresponding to the minimum singular valueσmin(Θ) and where the error εx is aligned with the right singular vector of Θ corresponding to themaximum singular value σmax(Θ). Thus, error is scaled by σmax while the signal is scaled by σmin

Θ (x + εx) = σminx + σmaxεx. (S7)

Thus, we see that the signal to noise ratio (SNR) is reduced by a factor equal to the condition number,

SNRin = xεx

=⇒ SNRout = σmin

σmax

xεx

= SNRin/κ. (S8)

If the condition number is large, then error can be amplified relative to the signal. The ideal conditionnumber is 1, where Θ has all singular values equal to 1, for example if Θ is a unitary matrix.Similarly, the sensitivity of the matrix inverse to error is also related to the condition number

x + εx = Θ−1 (y + εy) . (S9)

Again, worst-case scenario errors in the inversion are amplified by the condition number (this timewith the error being aligned with the singular vector corresponding to minimum singular value).In general, a random error εx or εy will have some component in this worst-case direction, and willbe amplified by the maximum singular value of Θ or Θ−1. Thus, it is desirable to explicitly controlthe condition number of Θ = CΨr by choice of the row selection operator C. For invertible matrices,the condition numbers of Θ and Θ−1 are the same. The discussion above generalizes for rectangularΘ.

is obtained using

x = Ψra, where a =

Θ−1y = (CΨr)−1y, p = r,

Θ†y = (CΨr)†y, p > r.

(15)A schematic of sparse sampling in a tailored basisΨr is shown in Fig. 3. The optimal sensor locationsare those that permit the best possible reconstruc-tion x. Thus, the sensor placement problem seeksrows of Ψr, corresponding to point sensor loca-tions in state space, that optimally condition inver-sion of the matrix Θ. For brevity in the followingdiscussion we denote the matrix to be invertedby Mγ = ΘTΘ (Mγ = Θ if p = r). Recall thatγ determines the structure of C, i.e. the sensorlocations, and hence affects the condition numbersof Θ and Mγ . The condition number of the sys-

tem may be indirectly bounded by optimizing thespectral content of Mγ using its determinant, trace,or spectral radius. For example, the spectral radiuscriterion for M−1

γ maximizes the smallest singularvalue of Mγ

γ? = argminγ,|γ|=p

‖M−1γ ‖2 = argmax

γ,|γ|=pσmin(Mγ). (16)

Likewise, the sum (trace) or product of magnitudes(determinant) of its eigenvalue or singular valuespectrum may be optimized

γ? = argmaxγ,|γ|=p

tr(Mγ,|γ|=p) = argmaxγ

∑i

λi(Mγ),

(17)

12

y

=C a

=a r ⇥

Figure 3: Full state reconstruction of x frompoint observations (y) is accomplished using leastsquares estimation of POD coefficients (a = Θ†y).

γ? = argmaxγ,|γ|=p

|det Mγ | = argmaxγ,|γ|=p

∏i

|λi(Mγ)|

= argmaxγ,|γ|=p

∏i

σi(Mγ). (18)

Direct optimization of the above criteria requiresa combinatorial search over

(np

)possible sensor

configurations and is hence computationally in-tractable even for moderate n. Several heuristicgreedy sampling methods have emerged for statereconstruction specifically with POD bases. Thesegappy POD [122] methods originally relied onrandom sub-sampling. However, significant per-formance advances where demonstrated by usingprincipled sampling strategies for reduced-ordermodels (ROMs) in fluid dynamics [123], oceanmodeling [124] and aerodynamics [125]. More re-cently, variants of the so-called empirical interpo-lation method (EIM, DEIM and Q-DEIM) [126],[127], [128] have provided near optimal samplingfor interpolative reconstruction of nonlinear termsin ROMs. This work examines an approximategreedy solution given by the matrix QR factoriza-tion with column pivoting of ΨT

r , which buildsupon the Q-DEIM method [128].

Sparse sensor placement with QR pivotingAn original contribution of this work is ex-

tending Q-DEIM to the case of oversampled sen-sor placement, where the number of sensors ex-ceeds the number of modes used in reconstruction(p > r). The key computational idea enabling

oversampling is the QR factorization with columnpivoting applied to the POD basis. QR pivotingitself dates back to the 1960s by Businger andGolub to solve least squares problems [129], and ithas found utility in various measurement selectionapplications [S2], [130], [131]. Similar to empiricalinterpolation methods such as DEIM, pivots fromthe QR factorization optimally condition the mea-surement or row selected POD basis, as describedbelow.

The reduced matrix QR factorization with col-umn pivoting decomposes a matrix A ∈ Rm×n

into a unitary matrix Q, an upper-triangular matrixR and a column permutation matrix C such thatACT = QR. The pivoting procedure provides anapproximate greedy solution method for the opti-mization in (18), which is also known as submatrixvolume maximization because matrix volume isthe absolute value of the determinant. QR columnpivoting increments the volume of the submatrixconstructed from the pivoted columns by selectinga new pivot column with maximal 2-norm, thensubtracting from every other column its orthogonalprojection onto the pivot column (see Algorithm2). Pivoting expands the submatrix volume byenforcing a diagonal dominance structure [128]

σ2i = |rii|2 ≥

k∑j=i|rjk|2; 1 ≤ i ≤ k ≤ m. (19)

This works because matrix volume is also theproduct of diagonal entries rii

|det A| =∏i

σi =∏i

|rii|. (20)

Furthermore, the oversampled case p > r maybe solved using the pivoted QR factorization ofΨrΨT

r , where the column pivots are selected fromn candidate state space locations based on theobservation that

det ΘTΘ =r∏i=1

σi(ΘΘT ), (21)

where we drop the absolute value since the deter-minant of ΘΘT is nonnegative.

13

Algorithm 1 Greedy sensor selection using a giventailored basis Ψr and number of sensors p

1: if p == r then2: γ ← qrPivot(Ψr, r)3: else if p > r then4: γ ← qrPivot(ΨrΨT

r , p)5: C←

[eγ1 eγ2 . . . eγp

]T

Sidebar: QR pivoting code

The QR pivoting procedure of Algorithm 1can be concisely implemented using the givenMatlab code. The pivoted QR factorization(Algorithm 2) is called within this code us-ing the native Matlab qr() subroutine. Thevariable Psi_r stores the given data-driventailored basis Ψr, and the desired list of sensorindices (QR pivots) is returned within thepivot variable.

if (p==r)% QR sensor selection, p=r (Q

-DEIM)[Q,R,pivot] = qr(Psi_r’,’

vector’);elseif (p>r)

% Oversampled QR sensors, p>r[Q,R,pivot] = qr(Psi_r*Psi_r

’,’vector’);endpivot = pivot(1:p)

Thus the QR factorization with column pivot-ing yields r point sensors (pivots) that best samplethe r basis modes Ψr

ΨTr CT = QR. (22)

Based on the same principle of pivoted QR, whichcontrols the condition number by minimizing thematrix volume, the oversampled case is handledby the pivoted QR factorization of ΨrΨT

r ,

(ΨrΨTr )CT = QR. (23)

Algorithm 2, the QR factorization, is natively im-plemented in most scientific computing softwarepackages. A Matlab code implementation of Al-gorithm 1 is provided in “Sidebar: QR pivotingcode". The oversampled case requires an expen-

sive QR factorization of an n × n matrix, whosestorage requirements scale quadratically with statedimension. However, this operation may be ad-vantageous for several reasons. The row selectiongiven by the first p QR pivots increase the lead-ing r singular values of ΘΘT , hence increasingdet ΘTΘ. This is the same maximization objectiveused in D-optimal experiment design [117], whichis typically solved with Newton iterations using aconvex relaxation of the subset selection objective.These methods require one matrix factorizationof an n × n matrix per iteration, leading to aruntime cost per iteration of at least O(n3). Theentire procedure must be recomputed for each newchoice of p. Our proposed method only requiresa single O(n3) QR factorization and results in ahierarchical list of all n total pivots, with the firstp pivots optimized for reconstruction in Ψr for anyp > r. Thus, additional sensors may be leveragedif available.

The QR factorization is implemented and op-timized in most standard scientific computingpackages and libraries, including Matlab, LA-PACK, NumPy, among many others. In addition tosoftware-enabled acceleration, QR runtime can besignificantly reduced by terminating the procedureafter the first p pivots are obtained. The operationcan be accelerated further using randomized tech-niques, for instance, by the random selection of thenext pivot [128] or by using random projections toselect blocks of pivots [132], [133], [134]. “Sidebar:QR pivoting fails to exploit sparsity" shows whyQR pivoting does not find the sparsest vector inan universal basis for compressed sensing.

The sparse sensor placement problem we haveposed here is related to machine learning conceptsof variable and feature selection [35], [135]. Suchsensor (feature) selection concepts generalize todata-driven classification. For image classificationusing linear discriminant analysis (LDA), sparsesensors may be selected that map via POD modesinto the discriminating subspace [34]. Moreover,sparse classification within libraries of POD modes

14

Algorithm 2 QR factorization with column pivoting of B ∈ Rn×m

1: procedure QRPIVOT( B, p )2: γ ← [ ]3: for k = 1, . . . , p do4: γk = argmaxj /∈γ ‖bj‖2

5: Find Householder Q such that Q ·

bkk

...bnk

=

?0...0

. ?’s are the diagonal entries of R

6: B← diag(Ik−1, Q) ·B remove from all columns the orthogonal projection onto bγk

7: γ ← [γ, γk]return γ

[136], [137], [40] can be improved by augmentingDEIM samples with a genetic algorithm [138] oradapting QR pivots for classification [57]. Sparsesensing has additionally been explored in signalprocessing for sampling and estimating signalsover graphs [139], [140], [141], [142].

Relation to optimal experimental designThe matrix volume objective described above

is closely related to D-optimal experiment de-sign [117]; in fact, the two problems are identicalwhen regarding the tailored basis Ψr as a setof n candidate experiments of a low-dimensionalsubspace. Classical experimental design selects thebest p out of n candidate experiments for es-timating r unknown parameters a ∈ Rr. Eachexperiment, denoted θi, produces one output yithat may include zero-mean i.i.d. Gaussian noiseξ ∼ N (0, η2). Again, we wish to estimate theparameters from p experiment outputs y ∈ Rp inthe following linear system,

y =

θ1

θ2...θp

·a+ξ =r∑

k=1

Ψγ1,k

Ψγ2,k

...Ψγp,k

ak+ξ = CΨra+ξ,

(24)which is equivalent to the state reconstruction for-mulation of gappy POD [122]. In Matlab notationwe sometimes refer to CΨr as Ψr(γ, :). Each pos-sible experiment θi may be regarded as a row of Θor of the tailored basis Ψr such that θi = Ψr(γi, :).Equivalently each θi is a weighted “measurement”

of the lower dimensional POD parameter space(not to be confused with the point measurementoperation C). Note that when all experiments areselected the output is simply the state vector xsince x = Ψra + ξ.

Given experiment selections indexed by γ, theestimation error covariance is given by

Var(a − a) = η2(ΘTΘ)−1 = η2((CΨr)TCΨr)−1.

(25)D-optimal subset selection minimizes the errorcovariance by maximizing the matrix volume ofMγ = ΘTΘ:

γ? = argmaxγ,|γ|=p

log detp∑i=1

θTi θi

= argmaxγ,|γ|=p

det(CΨr)TCΨr, (26)

which is equivalent to (18). Similarly, A-optimaland E-optimal design criteria optimize the traceand spectral radius of ΘTΘ, and are equivalentto (17) and (16), respectively. The exact solutionsof these optimization problems are intractable, andthey are generally solved using heuristics. Thisis most commonly accomplished by solving theconvex relaxation with a linear constraint on sensorweights β,

β? = argmaxβ∈Rn

log detn∑i=1

βiθTi θi,

subject ton∑i=1

0≤βi≤1

βi = p. (27)

15

Sidebar: QR pivoting fails to exploit sparsity

The QR pivoting algorithm can be used to efficiently find a sparse solution of an underdeterminedlinear system y = Θs. In fact, in Matlab, this is as simple as using the ‘\’ command:

>>s = Theta\y;

However, QR pivoting does not always provide the sparsest solution. In addition, the pivot locations,and hence the nonzero entries in s, are determined entirely by Θ. Therefore, the structure of thenonzero coefficients in s have nothing to do with the specific signal or measurements. Given p > Kmeasurements of an unknown signal in a universal basis, the QR pivot algorithm will fail in twoways: 1) it will return a sparse vector with p nonzero elements (instead of the desired K nonzeroelements), and 2) the nonzero elements will not depend on the frequency content of the actual signal.This observation is illustrated on two test images in Figure F3. Notice that the sparse coefficients forboth the mountain and cappuccino are the same, resulting in poor image reconstruction.

Mountain Cappuccino

True

0 1000 2000 3000 4000-10

0

10

20

30

s

0 1000 2000 3000 4000-20

0

20

40

s

Com

pres

sed

0 1000 2000 3000 4000-10

0

10

20

30

sthresh

0 1000 2000 3000 4000-10

0

10

20

30

sthresh

(a) Fourier coefficients. The compressed images are recovered from a very small percentage of nonzero Fouriercoefficients (red) that represent the spatial frequency signature of the image.

QR pivoting solution

p=K

1000 1050 1100 1150 1200-10

-5

0

5

10

1000 1050 1100 1150 1200-20

-10

0

10

20

p=

2000

1000 1050 1100 1150 1200-10

-5

0

5

10

1000 1050 1100 1150 1200-5

0

5

(b) Reconstruction. Even at nearly 50% pixel sampling, least squares regression using the universal Fourier basisis unable to recover the characteristic frequencies. This is because backslash sets the number of nonzero entriesin the solution to the number of samples. Additional constraints are needed to recover the true coefficients,which is possible with compressed sensing.

Figure F3

The optimized sensors are obtained by selectingthe largest sensor weights from β. The iterativemethods employed to solve this problem, i.e., con-vex optimization and semidefinite programs [1],

[2], require matrix factorizations of n×n matrices ineach iteration. Therefore they are computationallymore expensive than the QR pivoting methods,which cost one matrix factorization in total. Greedy

16

TABLE 1: Summary of sensor selection methods

Method Objective Runtime

D-optimal subset selectionγ? = argmax

γ,|γ|=plog det

p∑i=1

θTi θi

= argmaxγ,|γ|=p

det(CΨr)TCΨr

(np

)determinant evaluations

Convex optimization

β? = argmaxβ

log detn∑i=1

βiθTi θi

subject ton∑i=1

0≤βi≤1

βi = pO(n3) per iteration

QR pivoting (greedy) Case p = r : ΨTr CT = QR

Case p > r : (ΨrΨTr )CT = QR

O(nr2)O(n3)

sampling methods such as EIM and QR are prac-tical for sensor placement within a large numberof candidate locations in fine spatial grids; hence,they are the methods of choice in reduced-ordermodeling [143]. The various optimization methodsfor data-driven sensor selection are summarized inTable 1.

Comparison of methodsSensor selection and signal reconstruction al-

gorithms are implemented and compared on datafrom fluid dynamics, facial images, and oceansurface temperatures. The examples span a widerange of complexity, exhibit both rapid and slowsingular value decay, and come from both staticand dynamic systems.

In each example, optimized sensors obtainedin a tailored basis with QR pivots (See “Sparsesensor placement with QR pivoting") outperformrandom measurements in a universal basis usingcompressed sensing (See “Theory of compressedsensing") for signal reconstruction. Moreover, forthe same reconstruction performance, many fewerQR sensors are required, decreasing the cost asso-ciated with purchasing, placing, and maintainingsensors, as well as reducing the latency in compu-tations. Thus, for a well-scoped reconstruction taskwith sufficient training data, we advocate princi-pled sensor selection rather than compressed sens-ing. For example, the QR-based sampling method

is demonstrated with yet another tailored basiscommonly encountered in scientific computing –the Vandermonde matrix of polynomials. “Sidebar:Other tailored bases – polynomial interpolation"compares polynomial interpolation with QR pivotsfor the ill-conditioned set of equispaced points onan interval. When the structure of the underlyingsignal is unknown, then compressed sensing pro-vides more flexibility with an associated increasein the number of sensors.

Flow past a cylinderFluid flow past a stationary cylinder is a

canonical example in fluid dynamics that is high-dimensional yet reveals strongly periodic, low-rank phenomena. It is included here as an idealsystem for reduction via POD and hence, minimalsensor placement. The data is generated by nu-merical simulation of the linearized Navier-Stokesequations using the immersed boundary projec-tion method (IBPM) based on a fast multi-domainmethod [144], [145].

The computational domain consists of fournested grids so that the finest grid covers a domainof 9 × 4 cylinder diameters and the largest gridcovers a domain of 72× 32. Each grid has resolu-tion 450 × 200, and the simulation consists of 151timesteps with δt = 0.02. The Reynolds numberis 100, and the flow is characterized by laminarperiodic vortex shedding [146].

17

Sidebar: Other tailored bases – polynomial interpolation

Suppose x ∈ Rn is a suitably fine discretization of the interval [0, 1]. We may construct a degree rpolynomial interpolant of a function f(x) on this interval by forming the n× r Vandermonde basis

Ψr = [1 | x | x2 | . . . | xr−1]. (S10)

It is well-known that equispaced interpolation in the Vandermode basis is ill-conditioned; the mostcommonly used alternatives are Chebyshev or Legendre bases with non-equispaced points thatsatisfy a non-uniform density on the interval. Similar to the discussion above, we seek the (near)best interpolation samples (sensors) for interpolating arbitrary functions within this basis. This is anequivalent formulation to the above, except the basis under consideration is a tailored basis and nota data-driven one, however, the same sampling methodologies apply. The resulting samples, in thenon data-driven case, are general enough for sampling any well-behaved univariate function. Belowwe apply QR sampling on the polynomial basis functions to demonstrate the power of optimizedsampling.

f(x) =∣∣∣∣x2 − 1

2

∣∣∣∣ (S11)

This application of QR pivoting to find near-optimal Fekete points for polynomial interpolation wasfirst introduced in [S2]. Matlab implementation code for polynomial interpolation is listed below,and the comparison between equispaced and QR interpolation samples is shown in Figure F4.

r = 11; n = 1000;x = linspace(0,1,n+1)’;

% Construct Vandermonde matrix on[0,1]

Vde = zeros(n+1,r);for i = 1:r

Vde(:,i) = x.^(i-1);end

% approximate Fekete points[Q,R,pivot] = qr(Vde’,’vector’);fekete = pivot(1:r);

% equispaced pointsequi = 1:n/(r-1):(n+1);pts = fekete; color = ’b’;% build interpolantf = abs(x.^2-0.5);coefs = Vde(pts,:)\f(pts);pstar = Vde*coefs;

0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6Approximate Fekete points (QR pivots)

0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6Equispaced points

xFigure F4: Polynomial interpolation with se-lected QR pivots overcomes the Gibbs phe-nomenon observed with ill-conditioned equi-spaced points.

References[S2] A. Sommariva and M. Vianello, “Computing approximate Fekete points by QR factorizations of

Vandermonde matrices,", Computers & Mathematics with Applications, vol. 57, no. 8, pp. 1324–1336,2009.

Vorticity field snapshots are shown in Fig. 4.In the cylinder flow and sea surface temperatureexamples, each snapshot xi = x(ti) is a spatialmeasurement of the system at a given time ti. ThusPOD coefficients ak(ti) are time dependent, andψk(x) are spatial eigenmodes. The first 100 cylin-

der flow snapshots are used to train POD modesand QR sensors, and reconstruction error bars areplotted over 51 remaining validation snapshots inFigures F5b and 5.

18

Sidebar: Selecting number of sensors p and rank r for flow past a cylinder

An overarching goal of optimized sensor placement is choosing the fewest p sensors for reconstruc-tion. This sparse sensor optimization is facilitated by low-rank structure in the data (Fig. F5a) andinherently involves a trade-off between the number of sensors and reconstruction accuracy. As seenin Fig. F5b, effective sensor optimization moves the elbow of the reconstruction error curve downand to the left, indicating accurate reconstruction with few sensors. Reducing the number of sensorsmay be critically enabling when sensors are expensive or when low computational latency is desired.

0 20 40 60 80 100

10-8

10-7

10-6

10-5

10-4

10-3

10-2

10-1

100

Nor

mal

ized

sing

ular

valu

es

r

Mode 1Mode 42

Mode 43

(a) Singular value spectrum.

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95100

10-6

10-4

10-2

100

102

104

PODQR sensorsRandom sensors

Rec

onst

ruct

ion

erro

r

r

Goal: Fewer sensorsand lower error

(b) Reconstruction error with QR pivot sensors.

Figure F5: Illustration of singular value decay and sparse signal reconstruction in the fluid flow pasta cylinder example. (a) The exponential decay of singular values indicates low-rank dynamics witha modal truncation at r = 42 modes, the last modes containing any meaningful spatial structure. (b)Recovery with a minimal number of QR sensors with increasing POD basis rank is nearly as good asa snapshot’s approximation in the same POD basis, and overfitting occurs beyond r > 42. Optimizedsensing remains orders of magnitude more accurate than reconstruction with random sensors.

time

Snapshots

2

1

0

1

2

2

1

0

1

2

2

1

0

1

2

2

1

0

1

2

2

1

0

1

2

2

1

0

1

2x1

x2

xm

Figure 4: Fluid flow past a cylinder at Reynoldsnumber 100, characterized by vortex shedding.

The POD modes of this data reflect oscilla-tory dynamics characterized by periodic vortex-shedding. The data is low-rank, and the singularvalues decay rapidly, as shown in Fig. F5a. Thesingular values occur in pairs corresponding to

harmonics of the dominant vortex shedding fre-quency. Most of the spectral energy in the datasetis captured by the first 42 POD modes. Thus theintrinsic rank of the dataset in POD feature spaceis r = 42, and the minimal number of QR pivots isp = 42. This yields a dramatic reduction from theinitial state dimension of n ≈ 90000 spatial grid-points. Here, QR pivoting of ΨT

r (with O(nr2) op-erations) bypasses expensive O(n3) factorizationsof large n× n matrices with alternate sampling orconvex optimization methods.

Reconstruction from QR sensors (Fig. F5b) suc-cessfully captures modal content with only p = r

sensors when fitting to the first 42 POD modes. Thefirst 42 POD modes characterize nearly 100% of thesystem’s energy, the normalized sum of the singu-lar values. Using modes beyond r > 42 results in

19

0

0.026

316

0.052

632

0.078

947

0.10

526

0.13

158

0.15

789

0.18

421

0.21

053

0.23

684

0.26

316

0.28

947

0.31

579

0.34

211

0.36

842

0.39

474

0.42

105

0.44

737

0.47

368

0.

510-6

10-4

10-2

100

102

PODQR, p=2rQDEIMDEIMR

econ

stru

ctio

ner

ror

Sensor noise variance ηFigure 5: Comparison with DEIM. Oversampled QR permits additional sensors p = 2r and a 4x reductionin reconstruction error compared to discrete empirical interpolation method (DEIM). In the p = r = 40case DEIM and QR pivoting (Q-DEIM) perform comparably, with a slight improvement observed withQR. All sampling methods are within some constant factor of the optimal POD approximation with thefull states. Hence POD-based sampling methods demonstrate robust, bounded growth of reconstructionerror as sensor noise increases.

overfitting, and QR pivoting selects sensors basedon uninformative modes. Thus, accuracy stopsimproving beyond r = 42 target modes, where-upon sensor data amplifies noise. However, thesetailored sensors perform significantly better thanrandom sensors due to the favorable conditioningproperties of QR interpolation points. “Sidebar:Selecting number of sensors p and rank r for flowpast a cylinder" illustrates the ability of optimizedsensing to significantly reduce the number of sen-sors required for a given performance.

Noise comparison studyMeasurements of real-world data are often

corrupted by sensor noise. The POD-based sensorselection criteria, as well as A,D and E-optimal ex-perimental design criteria, are optimal for estima-tion with measurements corrupted by zero-meanGaussian white noise. We empirically demonstratethis on the cylinder flow data with increasing addi-tive white noise. Here we assume sensor noise onlyin the test measurements and not in the trainingdata or features, see Eqn. (24). The POD modes andthe different sensor sets are trained on the first 100snapshots, and these different sensor sets are used

to reconstruct the remaining 50 validation snap-shots, which were not used for training features.

The reconstruction accuracy of the varioussampling methods are compared for increasingsensor noise in Fig. 5, alongside the full-statePOD approximation for illustration. Here we trun-cate the POD expansion to r = 40 eigenmodesand compare the p = r reconstruction computedwith the discrete empirical interpolation method(DEIM) [127] against the QR pivoting reconstruc-tion (Q-DEIM, p = r). The DEIM greedy strategyplaces sensors at extrema of the residual com-puted from approximating the k-th mode with theprevious k − 1 sensors and eigenmodes. It canbe seen that QR reconstruction is slightly moreaccurate than that of DEIM, which is the leadingsampling method currently in use for reduced-order modeling [143].

QR pivoting is competitive in both speed andaccuracy. The speed of QR pivoting is enabled byseveral implementation accelerations; for example,the column norms in line 4 of Algorithm 2 are onlycomputed once and are then reused. Unlike QRpivoting, DEIM and related methods add succes-sive sensors per iteration by similarly optimizing

20

0 100 200 300 400 500 60010

-2

10-1

100

101

102

Mode 50

Mode 100

Mode 166Mode 300

r

σr

Figure 6: Normalized singular values and selected eigenfaces. Facial information progressively decreasesacross selected eigenfaces, and no facial features can be readily discerned beyond eigenface r = 166, theoptimal modal truncation value according to [121].

some metric over all locations. However, this met-ric (e.g., the approximation residual or the largestsingular value) is recomputed at every iteration.The QR factorization is significantly faster thanconvex optimization methods used in optimal de-sign of experiments, which typically require onematrix factorization per iteration.

In fact, convex optimization methods that re-lax the subset selection to weighted sensor place-ment provide no bounds for deviation from theglobal optimum, partly because rounding proce-dures are employed on the weights to decidethe final selection. However, reconstruction er-ror bounds for the globally optimal selection areknown for DEIM [127], Q-DEIM [128] and relatedPOD sampling methods [123], [124]. Furthermore,QR pivoting can achieve significant accuracy gainsover DEIM by oversampling – when p = 2r QRreconstruction error is 4x smaller than that ofDEIM. It should be noted that while DEIM hasnot yet been extended to the p > r case, oversam-pling methods exist for other POD-sampling meth-ods [123], [124]. However, the iterative proceduresinvolved in the latter are typically more expensive.Recent accelerated variants of greedy principledsampling [147] may permit oversampling for largen, when oversampled QR storage requirements

would be excessive. In the cylinder flow case,we bypass this storage requirement by uniformlydownsampling the fine grid by a factor of 5 ineach spatial direction, thus reducing the numberof candidate sensor locations to n = 3600 insteadof n = 89351.

Extended Yale B eigenfacesImage processing and computer vision com-

monly involve high-resolution data with dimen-sion determined by the number of pixels. Cam-eras and recording devices capture massive im-ages with rapidly increasing pixel and temporalresolution. However, most pixel information in animage can be discarded for subject identificationand automated decision-making tasks.

The extended Yale B face database [115], [116]is a canonical dataset used for facial recognition,and it is an ideal test bed for recovering low-rank structure from high-dimensional pixel space.The data consists of 64 aligned facial images eachof 38 stationary individuals in different lightingconditions. We validate our sensor (pixel) selectionstrategy by recovering missing pixel data in avalidation image using POD modes or eigenfacestrained on 32 randomly chosen images of eachindividual.

Normalized singular values are shown in

21

`2 QR sensors `2 Random sensors Compressed sensing (random)50 166 50 166 50 166 300 600

Sens

ors

Rec

onst

ruct

ion

Figure 7: Comparison of reconstruction and sensor selection methods. QR sensors emphasize facialfeatures such as the eyes, nose and mouth, and hence achieve adequate reconstruction with as few as 166sensors (16% of all pixels). Least squares (`2) reconstruction with 50 QR sensors (5% of pixels) surpassesthe performance of compressed sensing with 300 random pixels. In comparison, compressed sensingrequires 600 sampled pixels for comparable recovery, while `2 reconstruction with random sensors isconsistently poor.

Fig. 6, and the optimal singular value truncationthreshold [121] occurs at r = 166, indicating the in-trinsic rank of the training dataset. Indeed, selectedeigenfaces are also shown to reveal no meaningfulfacial structure beyond eigenface 166. QR pixelselection is performed on the first 50 and first 166eigenfaces, and selected pixels shown in Fig. 7cluster around important facial features – eyes,nose and mouth.

Image reconstructions in Fig. 7 are estimatedfrom the same number of selected pixels as thenumber of modes used for reconstruction. For in-stance, the 50 eigenface reconstruction is uniquelyconstructed from 50 selected pixels out of 1024total – 5% of available pixels. Even at lower pixelselection rates, least squares reconstruction fromQR selected pixels is more successful at filling inmissing data and recovering the subject’s face.

For comparison, reconstruction of the sameface from random pixels using compressed sens-ing is shown in Fig. 7. Compressed sensing ina universal Fourier basis demonstrates progres-sively improved global feature recovery. However,more than triple the pixels are required for thesame quality of reconstruction as in QR selection.

5 10 20 30 40 50 60 70 80 90 1000

20

40

60

80

100

PODConvex optQR, p=2rRandom sensors

r modes

Perc

ent

erro

r

Figure 8: Recovery using our oversampled QRpivoting method for p = 2r > r sensors. QRpivoting sensors perform comparably to sensorsobtained via the convex optimization (Convexopt) method [2], and both are close to the opti-mal proper orthogonal decomposition approxima-tion using the full state. In contrast to the con-vex method, QR sensors are obtained at reducedcomputational cost using the QR factorization ofΨrΨT

r . Both methods outperform an equal numberof randomly selected sensors.

Moreover, the convex `1 optimization procedureis extremely expensive compared to the single `2

regression on subsampled eigenfaces. Therefore

22

0 100 200 300 400 500 600 700 80010

0

101

102

103

104

105

Mode 100

Mode 200

Mode 302Mode 600

r

σr

Figure 9: Singular values and selected proper orthogonal decomposition (POD) modes for sea surfacetemperature data. The optimal rank truncation threshold occurs at r = 302. POD mode 302 containsenergetic localized convective phenomena (El Niño) but is largely uninformative on a global scale. Thus,overfitting may occur as more modes are included (see Fig. 11).

data-driven feature selection and structured mea-surement selection are of significant computationaland predictive benefit, and occur at the small train-ing expense of one SVD and pivoted QR operation.

The convergence of reconstruction with sen-sors using QR pivoting is shown in Fig. 8. Moresensors than modes are used in reconstructionfor this example. The expected error dropoff isobserved with increasing number of modes andsensors, although the dropoff is slower than forthe cylinder flow (Fig. F5b) due to slower decay ofsingular values.

Sea surface temperature (SST)Next we consider the NOAA_OISST_V2 global

ocean surface temperature dataset spanning theduration 1990–2016. The data is publicly availableonline [148]. Unlike eigenfaces, this dataset is atime series, for which a snapshot is recorded ev-ery week. Sensor selection must then track en-ergetic temporal signatures. Sensors and featuresare trained on the first 16 years (832 snapshots),and a test snapshot is selected from the excludedvalidation set. The singular values are shown inFig. 9.

Like the eigenfaces, localized convective phe-nomena have energetic contributions to otherwiseglobally uninformative eigenssts. This is best seen

in the POD snapshot projections, in which the 100eigensst projection already sufficiently recovers dy-namics, while increasing the number of eigensstsin the projection further refines convective phe-nomena. These lower-energy modes containingconvective effects contribute to some degree ofoverfitting in `2 reconstruction (Fig. 10). The mostinteresting of these is the El Niño southern os-cillation (ENSO) feature that is clearly identifiedfrom QR selected sensors. El Niño is defined asany temperature increase of a specified thresholdlasting greater than six months in this highlightedregion of the South Pacific. It has been implicatedin global weather patterns and climate change.

Remark: Modal separation of intermittent phe-nomena such as the El Niño is difficult from a time-invariant POD analysis. Separation of isolated,low-energy temporal events cannot be done froma variance-characterizing decomposition such asthe POD – reordering the snapshots will yield thesame dominant modes. On the other hand, tensordecompositions and temporal-frequency analysessuch as multiresolution dynamic mode decompo-sition have succeeded at identifying El Niño wherePOD has failed. Sensor selection using non-normalmodes arising from such decompositions remainsan open problem and the focus of ongoing work.

23

100 sensors 200 sensors 302 sensors

QR

Ran

dom

Figure 10: QR selected sensors used for reconstruction. QR sensors are informative about ocean dynamics,for example capturing convective phenomena such as the El Niño Southern Oscillation off coastal Peru.

100 mode approximation 200 mode approximation 302 mode approximation

POD

100 sensors 200 sensors 302 sensors

` 2Q

R` 2

Ran

dom

Com

pres

sed

Sens

ing

Figure 11: Comparison of methods for reconstruction of a single snapshot from sensors. QR selectedsensors filter uninformative features and achieve better reconstruction. In comparison, random sensorsachieve poor reconstruction with least squares (`2) and compressed sensing. As the number of properorthogonal decomposition (POD) modes is increased, modal approximation with the full state (top row)only gets marginally better, which indicates additional features contribute low-energy uninformativefeatures. Hence `2 reconstruction expresses low-energy POD modes and suffers from overfitting, withboth random and to a lesser extent, QR sensors.

24

Sidebar: Comparison – Sparse sensing methods

TABLE T1: Comparison: compressed sensing and data-driven sensing

Compressed sensing Data-driven QR sensingNo. samples O(K log n

K ) p ≥ r, r � KSamples Random OptimizedBasis Universal TailoredTraining data No YesSolution procedure Convex optimization Least squares

We summarize in Table T1 the distinction between two competing perspectives for signal recovery:compressed sensing and data-driven sensing in a tailored basis. Compressed sensing recoversan unknown, underdetermined signal, by sparsity-promoting `1 minimization. Figure F6 furtherillustrates the `q norm constraints (0 ≤ q ≤ 1), which promote sparsity of the solution. This approachmandates random measurements to maintain incoherence with the basis and requires O(K log n

K )measurements, where K is the signal’s sparsity within this basis. In contrast, data-driven sensingrequires only as many optimized samples as the data’s intrinsic rank r, where r is often much smallerthan K. Sensor locations are thus selected to be informative based on system structure, yieldingcompressed measurements that streamline subsequent analysis, particularly when the original systemis high-dimensional. Although compressed sensing can recover a wider range of signals, randomsensing and convex optimization procedures may be impractical for high-dimensional, structuredsignals encountered in physical systems. In these cases, data-driven sensing is beneficial since itpermits a drastic reduction in the required number of sensors and downstream computation.

a1

a2

`0 `1 `2 `∞

Figure F6: Progression of unit balls ‖a‖q = 1 of `q norm where q ∈ [0,∞]. Unit balls of`0, `1, `2, and `∞ norms are outlined in black. Minimizing `q norms when q ∈ [0, 1] promotes sparsitysince these unit balls are oriented toward the coordinate axes.

DiscussionThe efficient sensing of complex systems is an

important challenge across the physical, biological,social, and engineering sciences, with significantimplications for nearly all downstream tasks. Inthis work, we have demonstrated the practical im-plementation of several sparse sensing algorithmson a number of relevant real-world examples. Asdiscussed throughout, there is no all-purpose strat-egy for the sparse sensing of a high-dimensionalsystem. Instead, the choice depends on key factorssuch as the amount of training data available, thescope and focus of the desired estimation task,

cost constraints on the sensors themselves, andthe required latency of computations on sensordata. Thus, we partition the sparse sensing algo-rithms into two fundamental categories: 1) opti-mized sensing in a data-driven tailored basis, and2) random sensing in a universal basis.

A critical comparison of the two approacheshighlights a number of relative strengths andweaknesses. The first strategy results in a highlyoptimized set of sensors that are suitable for tightlyscoped reconstruction problems where sufficienttraining data is available. The second strategy re-quires more sensors for accurate reconstruction but

25

also makes fewer assumptions about the underly-ing signal, making it more general. We emphasizethat optimized sensing in a tailored basis typicallyprovides more accurate signal reconstruction thanrandom measurements, facilitating a reduction inthe number of sensors by about a factor of two.Further, sensor selection and signal reconstructionin the tailored basis is computationally efficientand simple to implement, while compressed sens-ing generally requires a costly iterative algorithm.In problems where sensors are expensive, or whenlow-latency decisions are required, the reductionin the number of sensors and the associated speed-up of optimized sensing can be significant. Thus,when the reconstruction task is well-scoped and asufficient volume of training data is available, weadvocate principled sensor selection rather thancompressed sensing. In addition, pivoted QR sen-sors may be used in conjunction with other tailoredbases (polynomials, radial basis functions) whensignal structure is known. Since these are not data-driven basis functions, QR optimized samples cangeneralize to different dynamical regimes or flowgeometries.

Potential applied impactMany fields in science and engineering rely on

sensing and imaging. Moreover, any applicationinvolving feedback control for stabilization, perfor-mance enhancement, or disturbance rejection reliescritically on the choice of sensors. We may roughlycategorize these sensor-critical problems into twobroad categories: 1) problems where sensors areexpensive and few (ocean sampling, disease mon-itoring, espionage, etc.), and 2) problems wheresensors are cheap and abundant (cameras, high-performance computation, etc.).

In the first category, where sensors come at ahigh cost, the need for optimized sparse sensorsis clear. However, it is not always obvious how tocollect the training data required to optimize thesesensors. In some applications, high-fidelity simula-tions may provide insight into coherent structures,whereas in other cases a large-scale survey may be

required. It has recently been shown that it maybe possible to optimize sensors based on heavilysubsampled data, as long as coherent structures arenon-localized [34].

In the second category, where sensors arereadily available, it may still be advantageousto identify key sensors for fast control deci-sions. For example, in mobile applications, suchas vision-based control of a quad-rotor or un-derwater monitoring of an energy harvesting siteusing an autonomous underwater vehicle, com-putational and battery resources may be limited.Restricting high-dimensional measurements to asmall subset of key pixels speeds up compu-tation and reduces power consumption. Similarperformance enhancements are already exploitedin high-performance computing, where expensivefunction evaluations are avoided by sampling atkey interpolation points [149], [127]. Finally, itmay also be the case that if measurements arecorrupted by noise, reconstruction may improveif uninformative sensors are selectively ignored.

Reproducible ResearchA Matlab code supplement is available [8] for

reproducing results in this manuscript, including:

1) Datasets in Matlab file formats, or links todata that are publicly available online;

2) Matlab scripts to recreate figures of results.

References

[1] S. Boyd and L. Vandenberghe, Convex opti-mization. Cambridge university press, 2004.

[2] S. Joshi and S. Boyd, “Sensor selection viaconvex optimization,” IEEE Transactions onSignal Processing, vol. 57, no. 2, pp. 451–462,2009.

[3] W. F. Caselton and J. V. Zidek, “Opti-mal monitoring network designs,” Statis-tics &amp; Probability Letters, vol. 2, no. 4,pp. 223–227, 1984.

[4] A. Krause, A. Singh, and C. Guestrin, “Near-optimal sensor placements in gaussian pro-cesses: Theory, efficient algorithms and em-

26

pirical studies,” Journal of Machine LearningResearch, vol. 9, no. Feb, pp. 235–284, 2008.

[5] D. V. Lindley, “On a measure of the informa-tion provided by an experiment,” The Annalsof Mathematical Statistics, pp. 986–1005, 1956.

[6] P. Sebastiani and H. P. Wynn, “Maximumentropy sampling and optimal bayesian ex-perimental design,” Journal of the Royal Statis-tical Society: Series B (Statistical Methodology),vol. 62, no. 1, pp. 145–157, 2000.

[7] L. Paninski, “Asymptotic theory ofinformation-theoretic experimental design,”Neural Computation, vol. 17, no. 7, pp. 1480–1507, 2005.

[8] K. Manohar, B. W. Brunton, J. N. Kutz, andS. L. Brunton, “Code supplement to "Data-driven sparse sensor placement".” https://github.com/kmanohar/SSPOR_pub, Jan-uary 2017.

[9] S. L. Brunton and B. R. Noack, “Closed-loop turbulence control: Progress and chal-lenges,” Applied Mechanics Reviews, vol. 67,pp. 050801–1–050801–48, 2015.

[10] Y. Susuki and I. Mezic, “Nonlinear Koop-man modes and a precursor to power sys-tem swing instabilities,” IEEE Transactions onPower Systems, vol. 27, no. 3, pp. 1182–1191,2012.

[11] J. L. Proctor and P. A. Eckhoff, “Discover-ing dynamic patterns from infectious diseasedata using dynamic mode decomposition,”International health, vol. 7, no. 2, pp. 139–145,2015.

[12] R. M. Plenge, E. M. Scolnick, andD. Altshuler, “Validating therapeutic targetsthrough human genetics,” Nature PublishingGroup, vol. 12, no. 8, pp. 581–594, 2013.

[13] K. Carlberg, C. Bou-Mosleh, and C. Farhat,“Efficient non-linear model reduction via aleast-squares Petrov–Galerkin projection andcompressive tensor approximations,” Inter-national Journal for Numerical Methods in En-gineering, vol. 86, no. 2, pp. 155–181, 2011.

[14] K. Carlberg, C. Farhat, J. Cortial, and D. Am-sallem, “The GNAT method for nonlinearmodel reduction: effective implementationand application to computational fluid dy-namics and turbulent flows,” Journal of Com-putational Physics, vol. 242, pp. 623–647, 2013.

[15] K. K. Chen and C. W. Rowley, “H2 optimalactuator and sensor placement in the lin-earised complex Ginzburg-Landau system,”Journal of Fluid Mechanics, vol. 681, pp. 241–260, 2011.

[16] J. N. Kutz, Data-Driven Modeling & ScientificComputation: Methods for Complex Systems &Big Data. Oxford University Press, 2013.

[17] L. Sirovich, “Turbulence and the dynamicsof coherent structures, parts I-III,” Q. Appl.Math., vol. XLV, no. 3, pp. 561–590, 1987.

[18] G. Berkooz, P. Holmes, and J. L. Lumley,“The proper orthogonal decomposition inthe analysis of turbulent flows,” Annual re-view of fluid mechanics, vol. 25, no. 1, pp. 539–575, 1993.

[19] P. J. Holmes, J. L. Lumley, G. Berkooz, andC. W. Rowley, Turbulence, coherent structures,dynamical systems and symmetry. CambridgeMonographs in Mechanics, Cambridge, Eng-land: Cambridge University Press, 2nd ed.,2012.

[20] P. J. Schmid, “Dynamic mode decompositionof numerical and experimental data,” Journalof Fluid Mechanics, vol. 656, pp. 5–28, Aug.2010.

[21] C. W. Rowley, I. Mezic, S. Bagheri, P. Schlat-ter, and D. Henningson, “Spectral analysisof nonlinear flows,” J. Fluid Mech., vol. 645,pp. 115–127, 2009.

[22] J. H. Tu, C. W. Rowley, D. M. Luchten-burg, S. L. Brunton, and J. N. Kutz, “Ondynamic mode decomposition: theory andapplications,” Journal of Computational Dy-namics, vol. 1, no. 2, pp. 391–421, 2014.

[23] J. N. Kutz, S. L. Brunton, B. W. Brunton,and J. L. Proctor, Dynamic Mode Decompo-

27

sition: Data-Driven Modeling of Complex Sys-tems. SIAM, 2016.

[24] R. R. Coifman, S. Lafon, A. B. Lee, M. Mag-gioni, B. Nadler, F. Warner, and S. W. Zucker,“Geometric diffusions as a tool for harmonicanalysis and structure definition of data:Diffusion maps,” Proceedings of the NationalAcademy of Sciences of the United States ofAmerica, vol. 102, no. 21, pp. 7426–7431, 2005.

[25] R. R. Coifman and S. Lafon, “Diffusionmaps,” Applied and computational harmonicanalysis, vol. 21, no. 1, pp. 5–30, 2006.

[26] R. R. Coifman, I. G. Kevrekidis, S. La-fon, M. Maggioni, and B. Nadler, “Diffu-sion maps, reduction coordinates, and lowdimensional representation of stochastic sys-tems,” Multiscale Modeling &amp; Simulation,vol. 7, no. 2, pp. 842–864, 2008.

[27] B. C. Moore, “Principal component analy-sis in linear systems: Controllability, observ-ability, and model reduction,” IEEE Transac-tions on Automatic Control, vol. AC-26, no. 1,pp. 17–32, 1981.

[28] K. Willcox and J. Peraire, “Balanced modelreduction via the proper orthogonal decom-position,” AIAA Journal, vol. 40, no. 11,pp. 2323–2330, 2002.

[29] C. W. Rowley, “Model reduction for fluidsusing balanced proper orthogonal decompo-sition.,” International Journal of Bifurcation andChaos, vol. 15, no. 3, pp. 997–1013, 2005.

[30] J. N. Juang and R. S. Pappa, “An eigensystemrealization algorithm for modal parameteridentification and model reduction,” Journalof Guidance, Control, and Dynamics, vol. 8,no. 5, pp. 620–627, 1985.

[31] E. J. Candès, J. Romberg, and T. Tao, “Stablesignal recovery from incomplete and inaccu-rate measurements,” Communications in Pureand Applied Mathematics, vol. 8, no. 1207–1223, 59.

[32] D. L. Donoho, “Compressed sensing,” IEEETransactions on Information Theory, vol. 52,

no. 4, pp. 1289–1306, 2006.

[33] R. G. Baraniuk, “Compressive sensing,” IEEESignal Processing Magazine, vol. 24, no. 4,pp. 118–120, 2007.

[34] B. W. Brunton, S. L. Brunton, J. L. Proctor,and J. N. Kutz, “Sparse sensor placementoptimization for classification,” SIAM Jour-nal on Applied Mathematics, vol. 76, no. 5,pp. 2099–2122, 2016.

[35] R. Tibshirani, “Regression shrinkage and se-lection via the lasso,” Journal of the RoyalStatistical Society. Series B (Methodological),pp. 267–288, 1996.

[36] T. Hastie, R. Tibshirani, and J. Friedman, Theelements of statistical learning, vol. 2. Springer,2009.

[37] G. James, D. Witten, T. Hastie, and R. Tib-shirani, An introduction to statistical learning.Springer, 2013.

[38] H. Schaeffer, R. Caflisch, C. D. Hauck, andS. Osher, “Sparse dynamics for partial differ-ential equations,” Proceedings of the NationalAcademy of Sciences USA, vol. 110, no. 17,pp. 6634–6639, 2013.

[39] V. Ozolin, š, R. Lai, R. Caflisch, and S. Osher,“Compressed modes for variational prob-lems in mathematics and physics,” Pro-ceedings of the National Academy of Sciences,vol. 110, no. 46, pp. 18368–18373, 2013.

[40] S. L. Brunton, J. H. Tu, I. Bright, andJ. N. Kutz, “Compressive sensing and low-rank libraries for classification of bifurcationregimes in nonlinear dynamical systems,”SIAM Journal on Applied Dynamical Systems,vol. 13, no. 4, pp. 1716–1732, 2014.

[41] J. L. Proctor, S. L. Brunton, B. W. Brunton,and J. N. Kutz, “Exploiting sparsity andequation-free architectures in complex sys-tems (invited review),” The European Phys-ical Journal Special Topics, vol. 223, no. 13,pp. 2665–2684, 2014.

[42] S. L. Brunton, J. L. Proctor, and J. N. Kutz,“Discovering governing equations from data

28

by sparse identification of nonlinear dy-namical systems,” Proceedings of the NationalAcademy of Sciences, vol. 113, no. 15, pp. 3932–3937, 2016.

[43] N. M. Mangan, S. L. Brunton, J. L. Proc-tor, and J. N. Kutz, “Inferring biologicalnetworks by sparse identification of nonlin-ear dynamics,” IEEE Transactions on Molecu-lar, Biological, and Multi-Scale Communications,vol. 2, no. 1, pp. 52–63, 2016.

[44] J.-C. Loiseau and S. L. Brunton, “Constrainedsparse Galerkin regression,” arXiv preprintarXiv:1611.03271, 2016.

[45] S. H. Rudy, S. L. Brunton, J. L. Proctor, andJ. N. Kutz, “Data-driven discovery of par-tial differential equations,” Science Advances,vol. 3, no. e1602614, 2017.

[46] T. Duriez, S. L. Brunton, and B. R. Noack,Machine Learning Control: Taming NonlinearDynamics and Turbulence. Springer, 2016.

[47] H. Schaeffer, “Learning partial differentialequations via data discovery and sparse op-timisation,” in Proc. R. Soc. A., vol. 473,p. 20160446, The Royal Society, 2017.

[48] Z. Bai, T. Wimalajeewa, Z. Berger, G. Wang,M. Glauser, and P. K. Varshney, “Low-dimensional approach for reconstruction ofairfoil data via compressive sensing,” AIAAJournal, vol. 53, no. 4, pp. 920–933, 2014.

[49] S. L. Brunton, J. L. Proctor, J. H. Tu, andJ. N. Kutz, “Compressed sensing and dy-namic mode decomposition,” Journal of Com-putational Dynamics, vol. 2, no. 2, pp. 165–191,2015.

[50] J. H. Tu, C. W. Rowley, J. N. Kutz, and J. K.Shang, “Spectral analysis of fluid flows usingsub-Nyquist-rate PIV data,” Experiments inFluids, vol. 55, no. 9, pp. 1–13, 2014.

[51] M. R. Jovanovic, P. J. Schmid, and J. W.Nichols, “Sparsity-promoting dynamic modedecomposition,” Physics of Fluids, vol. 26,no. 2, p. 024103, 2014.

[52] F. Gueniat, L. Mathelin, and L. Pastur,

“A dynamic mode decomposition approachfor large and arbitrarily sampled systems,”Physics of Fluids, vol. 27, no. 2, p. 025113,2015.

[53] H. Akaike, “Information theory and an ex-tension of the maximum likelihood princi-ple,” in Petrov, B.N.; Csáki, F., 2nd InternationalSymposium on Information Theory, Tsahkadsor,Armenia, USSR, September 2-8, 1971,, pp. 267–281, Budapest: Akadémiai Kiadó, 1973.

[54] G. Schwarz, “Estimating the dimension ofa model,” The Annals of Statistics, no. 6,pp. 461–464, 1978.

[55] D. Giannakis, A. J. Majda, and I. Horenko,“Information theory, model error, and pre-dictive skill of stochastic models for com-plex nonlinear systems,” Physica D, vol. 241,no. 20, pp. 1735–1752, 2012.

[56] N. M. Mangan, J. N. Kutz, S. L. Brunton, andJ. L. Proctor, “Model selection for dynamicalsystems via sparse regression and informa-tion criteria,” arXiv preprint arXiv:1701.01773,2017.

[57] K. Manohar, S. L. Brunton, and J. N. Kutz,“Environmental identification in flight usingsparse approximation of wing strain,” Jour-nal of Fluids and Structures, vol. 70, pp. 162–180, 2017.

[58] S. Skogestad and I. Postlethwaite, Multivari-able feedback control: analysis and design. Hobo-ken, New Jersey: John Wiley & Sons, Inc.,2 ed., 2005.

[59] G. E. Dullerud and F. Paganini, A course inrobust control theory: A convex approach. Textsin Applied Mathematics, Berlin, Heidelberg:Springer, 2000.

[60] P. Holmes and J. Guckenheimer, Nonlinearoscillations, dynamical systems, and bifurcationsof vector fields, vol. 42 of Applied MathematicalSciences. Springer, 1983.

[61] R. E. Kalman, “A new approach to linearfiltering and prediction problems,” Journal ofFluids Engineering, vol. 82, no. 1, pp. 35–45,

29

1960.

[62] G. Welch and G. Bishop, “An introduction tothe Kalman filter,” 1995.

[63] L. Ljung, System Identification: Theory for theUser. Prentice Hall, 1999.

[64] J. N. Juang, M. Phan, L. G. Horta, andR. W. Longman, “Identification of ob-server/Kalman filter Markov parameters:Theory and experiments,” Technical Memo-randum 104069, NASA, 1991.

[65] J. N. Juang, Applied System Identification. Up-per Saddle River, New Jersey: Prentice HallPTR, 1994.

[66] J. C. Doyle, “Guaranteed margins for LQGregulators,” IEEE Transactions on AutomaticControl, vol. 23, no. 4, pp. 756–757, 1978.

[67] J. C. Doyle and G. Stein, “Multivariable feed-back design: concepts for a classical/modernsynthesis,” IEEE Transactions on AutomaticControl, vol. 26, no. 1, pp. 4–16, 1981.

[68] N. E. Leonard and E. Fiorelli, “Virtual lead-ers, artificial potentials and coordinated con-trol of groups,” in Decision and Control, 2001.Proceedings of the 40th IEEE Conference on,vol. 3, pp. 2968–2973, IEEE, 2001.

[69] R. Olfati-Saber and R. M. Murray, “Consen-sus problems in networks of agents withswitching topology and time-delays,” Auto-matic Control, IEEE Transactions on, vol. 49,no. 9, pp. 1520–1533, 2004.

[70] J. C. Doyle, D. L. Alderson, L. Li, S. Low,M. Roughan, S. Shalunov, R. Tanaka, andW. Willinger, “The “robust yet fragile” natureof the internet,” Proceedings of the NationalAcademy of Sciences of the United States ofAmerica, vol. 102, no. 41, pp. 14497–14502,2005.

[71] N. E. Leonard, D. A. Paley, F. Lekien,R. Sepulchre, D. M. Fratantoni, and R. E.Davis, “Collective motion, sensor networks,and ocean sampling,” Proceedings of the IEEE,vol. 95, no. 1, pp. 48–74, 2007.

[72] A. Rahmani, M. Ji, M. Mesbahi, and

M. Egerstedt, “Controllability of multi-agentsystems from a graph-theoretic perspective,”SIAM J. on Control and Optimization, vol. 48,no. 1, pp. 162–186, 2009.

[73] B. O. Koopman, “Hamiltonian systems andtransformation in Hilbert space,” Proceedingsof the National Academy of Sciences, vol. 17,no. 5, pp. 315–318, 1931.

[74] I. Mezic, “Spectral properties of dynamicalsystems, model reduction and decomposi-tions,” Nonlinear Dynamics, vol. 41, no. 1-3,pp. 309–325, 2005.

[75] M. Budišic and I. Mezic, “An approximateparametrization of the ergodic partition us-ing time averaged observables,” in Decisionand Control, 2009 held jointly with the 2009 28thChinese Control Conference. CDC/CCC 2009.Proceedings of the 48th IEEE Conference on,pp. 3162–3168, IEEE, 2009.

[76] M. Budišic, R. Mohr, and I. Mezic, “AppliedKoopmanism a),” Chaos: An InterdisciplinaryJournal of Nonlinear Science, vol. 22, no. 4,p. 047510, 2012.

[77] I. Mezic, “Analysis of fluid flows via spectralproperties of the Koopman operator,” AnnualReview of Fluid Mechanics, vol. 45, pp. 357–378, 2013.

[78] M. O. Williams, C. W. Rowley, I. Mezic, andI. G. Kevrekidis, “Data fusion via intrinsicdynamic variables: An application of data-driven Koopman spectral analysis,” EPL (Eu-rophysics Letters), vol. 109, no. 4, p. 40007,2015.

[79] A. Surana, “Koopman operator based ob-server synthesis for control-affine nonlinearsystems,” in 55th IEEE Conference on Decisionand Control (CDC), pp. 6492–6499, IEEE, 2016.

[80] A. Surana and A. Banaszuk, “Linear ob-server synthesis for nonlinear systems us-ing Koopman operator framework,” IFAC-PapersOnLine, vol. 49, no. 18, pp. 716 – 723,2016. 10th IFAC Symposium on NonlinearControl Systems NOLCOS 2016.

30

[81] S. L. Brunton, B. W. Brunton, J. L. Proc-tor, and J. N. Kutz, “Koopman observablesubspaces and finite linear representationsof nonlinear dynamical systems for control,”PLoS ONE, vol. 11, no. 2, p. e0150171, 2016.

[82] J. L. Proctor, S. L. Brunton, and J. N. Kutz,“Dynamic mode decomposition with con-trol,” SIAM Journal on Applied Dynamical Sys-tems, vol. 15, no. 1, pp. 142–161, 2016.

[83] J. L. Proctor, S. L. Brunton, and J. N.Kutz, “Generalizing Koopman theory to al-low for inputs and control,” arXiv preprintarXiv:1602.07647, 2016.

[84] M. Korda and I. Mezic, “Linear predictorsfor nonlinear dynamical systems: Koopmanoperator meets model predictive control,”arXiv preprint arXiv:1611.03537, 2016.

[85] D. Giannakis, “Data-driven spectral decom-position and forecasting of ergodic dynami-cal systems,” arXiv preprint arXiv:1507.02338,2015.

[86] S. L. Brunton, B. W. Brunton, J. L. Proctor,E. Kaiser, and J. N. Kutz, “Chaos as anintermittently forced linear system,” NatureCommunications, vol. 8, no. 19, pp. 1–9, 2017.

[87] F. Takens, “Detecting strange attractors inturbulence,” Lecture Notes in Mathematics,vol. 898, pp. 366–381, 1981.

[88] A. J. Majda, J. Harlim, and B. Gershgorin,“Mathematical strategies for filtering turbu-lent dynamical systems,” Discrete and Con-tinuous Dynamical Systems, vol. 27, no. 2,pp. 441–486, 2010.

[89] J. Osth, B. R. Noack, and S. Krajnovic, “Onthe need for a nonlinear subscale turbulenceterm in POD models as exemplified for ahigh Reynolds number flow over an Ahmedbody,” Journal of Fluid Mechanics, vol. 747,pp. 518–544, 2014.

[90] E. Kaiser, B. R. Noack, L. Cordier, A. Spohn,M. Segond, M. Abel, G. Daviller, J. Osth,S. Krajnovic, and R. K. Niven, “Cluster-basedreduced-order modelling of a mixing layer,”

J. Fluid Mech., vol. 754, pp. 365–414, 2014.

[91] A. G. Nair and K. Taira, “Network-theoreticapproach to sparsified discrete vortex dy-namics,” Journal of Fluid Mechanics, vol. 768,pp. 549–571, 2015.

[92] K. Taira, A. G. Nair, and S. L. Brunton,“Network structure of two-dimensional de-caying isotropic turbulence,” Journal of FluidMechanics, vol. 795, p. R2, 2016.

[93] B. R. Noack, W. Stankiewicz, M. Morzynski,and P. J. Schmid, “Recursive dynamic modedecomposition of a transient cylinder wake,”Journal of Fluid Mechanics, vol. 809, pp. 843–872, 2016.

[94] G. Tissot, L. Cordier, N. Benard, and B. R.Noack, “Model reduction using dynamicmode decomposition,” Comptes Rendus Mé-canique, 2014.

[95] D. Amsallem, M. J. Zahr, and C. Farhat,“Nonlinear model order reduction basedon local reduced-order bases,” InternationalJournal for Numerical Methods in Engineering,vol. 92, no. 10, pp. 891–916, 2012.

[96] J. N. Kutz, X. Fu, and S. L. Brunton, “Multi-resolution dynamic mode decomposition,”SIAM Journal on Applied Dynamical Systems,vol. 15, no. 2, pp. 713–735, 2016.

[97] E. J. Candès and T. Tao, “Near optimal signalrecovery from random projections: Universalencoding strategies?,” IEEE Transactions onInformation Theory, vol. 52, no. 12, pp. 5406–5425, 2006.

[98] E. J. Candès, J. Romberg, and T. Tao, “Robustuncertainty principles: exact signal recon-struction from highly incomplete frequencyinformation,” IEEE Transactions on Informa-tion Theory, vol. 52, no. 2, pp. 489–509, 2006.

[99] E. J. Candès, “Compressive sensing,” Pro-ceedings of the International Congress of Math-ematics, 2006.

[100] E. J. Candès and M. B. Wakin, “An introduc-tion to compressive sampling,” IEEE SignalProcessing Magazine, pp. 21–30, 2008.

31

[101] R. G. Baraniuk, V. Cevher, M. F. Duarte, andC. Hegde, “Model-based compressive sens-ing,” IEEE Transactions on Information Theory,vol. 56, no. 4, pp. 1982–2001, 2010.

[102] D. L. Donoho, “For most large underdeter-mined systems of linear equations the min-imal `1-norm solution is also the sparsestsolution,” Communications on pure and appliedmathematics, vol. 59, no. 6, pp. 797–829, 2006.

[103] E. J. Candes and T. Tao, “Decoding by lin-ear programming,” Information Theory, IEEETransactions on, vol. 51, no. 12, pp. 4203–4215,2005.

[104] E. J. Candès, Y. Eldar, D. Needell, andP. Randall, “Compressed sensing with co-herent and redundant dictionaries,” Appliedand Computational Harmonic Analysis, vol. 31,pp. 59–73, 2010.

[105] J. A. Tropp, “Greed is good: Algorithmic re-sults for sparse approximation,” IEEE Trans-actions on Information Theory, vol. 50, no. 10,pp. 2231–2242, 2004.

[106] J. A. Tropp, “Just relax: Convex program-ming methods for identifying sparse signalsin noise,” IEEE Transactions on InformationTheory, vol. 52, no. 3, pp. 1030–1051, 2006.

[107] J. A. Tropp, A. C. Gilbert, and M. J. Strauss,“Algorithms for simultaneous sparse ap-proximation. part i: Greedy pursuit,” SignalProcessing, vol. 86, no. 3, pp. 572–588, 2006.

[108] J. A. Tropp and A. C. Gilbert, “Signal recov-ery from random measurements via orthog-onal matching pursuit,” IEEE Transactions onInformation Theory, vol. 53, no. 12, pp. 4655–4666, 2007.

[109] A. C. Gilbert and P. Indyk, “Sparse recov-ery using sparse matrices,” Proceedings of theIEEE, vol. 98, no. 6, pp. 937–947, 2010.

[110] D. Needell and J. A. Tropp, “CoSaMP: iter-ative signal recovery from incomplete andinaccurate samples,” Communications of theACM, vol. 53, no. 12, pp. 93–100, 2010.

[111] P. Li, T. J. Hastie, and K. W. Church, “Very

sparse random projections,” in Proceedings ofthe 12th ACM SIGKDD international confer-ence on Knowledge discovery and data mining,pp. 287–296, ACM, 2006.

[112] W. Wang, M. Wainwright, and K. Ramchan-dran, “Information-theoretic limits on sparsesignal recovery: Dense versus sparse mea-surement matrices,” Information Theory, IEEETransactions on, vol. 56, no. 6, pp. 2967–2979,2010.

[113] H. Nyquist, “Certain topics in telegraphtransmission theory,” Transactions of theAmerican Institute of Electrical Engineers,vol. 47, no. 2, pp. 617–644, 1928.

[114] C. E. Shannon, “A mathematical theory ofcommunication,” Bell System Technical Jour-nal, vol. 27, no. 3, pp. 379–423, 1948.

[115] A. Georghiades, P. Belhumeur, and D. Krieg-man, “From few to many: Illumination conemodels for face recognition under variablelighting and pose,” IEEE Trans. Pattern Anal.Mach. Intelligence, vol. 23, no. 6, pp. 643–660,2001.

[116] K. Lee, J. Ho, and D. Kriegman, “Acquiringlinear subspaces for face recognition undervariable lighting,” IEEE Trans. Pattern Anal.Mach. Intelligence, vol. 27, no. 5, pp. 684–698,2005.

[117] P. Y. Papalambros and D. J. Wilde, Principlesof optimal design: modeling and computation.Cambridge university press, 2000.

[118] K. Pearson, “On lines and planes of closest fitto systems of points in space.,” PhilosophicalMagazine, vol. 2, no. 7–12, pp. 559–572, 1901.

[119] E. N. Lorenz, “Empirical orthogonal func-tions and statistical weather prediction,” Sta-tistical Forecasting Project, Massachusetts In-stitute of Technology, Department of Meteo-rology, 1956.

[120] C. Eckart and G. Young, “The approximationof one matrix by another of lower rank,”Psychometrika, vol. 1, no. 3, pp. 211–218, 1936.

[121] M. Gavish and D. L. Donoho, “The opti-

32

mal hard threshold for singular values is4/√

3,” IEEE Transactions on Information The-ory, vol. 60, no. 8, pp. 5040–5053, 2014.

[122] R. Everson and L. Sirovich, “Karhunen–loèveprocedure for gappy data,” Journal of theOptical Society of America A, vol. 12, no. 8,pp. 1657–1664, 1995.

[123] K. Willcox, “Unsteady flow sensing and es-timation via the gappy proper orthogonaldecomposition,” Computers & Fluids, vol. 35,no. 2, pp. 208–226, 2006.

[124] B. Yildirim, C. Chryssostomidis, and G. E.Karniadakis, “Efficient sensor placement forocean measurements using low-dimensionalconcepts,” Ocean Modelling, vol. 27, no. 3,pp. 160–173, 2009.

[125] H. Chen, Q. Wang, R. Hu, and P. Constan-tine, “Conditional sampling and experimentdesign for quantifying manufacturing errorof transonic airfoil,” in 49th AIAA AerospaceSciences Meeting including the New HorizonsForum and Aerospace Exposition, p. 658, 2011.

[126] M. Barrault, Y. Maday, N. C. Nguyen, andA. T. Patera, “An ‘empirical interpolation’method: application to efficient reduced-basis discretization of partial differentialequations,” Comptes Rendus Mathematique,vol. 339, no. 9, pp. 667–672, 2004.

[127] S. Chaturantabut and D. C. Sorensen, “Non-linear model reduction via discrete empiri-cal interpolation,” SIAM Journal on ScientificComputing, vol. 32, no. 5, pp. 2737–2764,2010.

[128] Z. Drmac and S. Gugercin, “A new selectionoperator for the discrete empirical interpola-tion method—improved a priori error boundand extensions,” SIAM Journal on ScientificComputing, vol. 38, no. 2, pp. A631–A648,2016.

[129] P. Businger and G. H. Golub, “Linear leastsquares solutions by Householder trans-formations,” Numerische Mathematik, vol. 7,no. 3, pp. 269–276, 1965.

[130] L. P. Heck, J. Olkin, and K. Naghshineh,“Transducer placement for broadband activevibration control using a novel multidimen-sional QR factorization,” Journal of vibrationand acoustics, vol. 120, no. 3, pp. 663–670,1998.

[131] P. Seshadri, A. Narayan, and S. Mahadevan,“Optimal quadrature subsampling for leastsquares polynomial approximations,” arXivpreprint arXiv:1601.05470, 2016.

[132] P.-G. Martinsson, G. Quintana Ortí,N. Heavner, and R. van de Geijn,“Householder QR factorization withrandomization for column pivoting(HQRRP),” SIAM Journal on ScientificComputing, vol. 39, no. 2, pp. C96–C115,2017.

[133] P.-G. Martinsson, “Blocked rank-revealingQR factorizations: How randomized sam-pling can be used to avoid single-vector piv-oting,” arXiv preprint arXiv:1505.08115, 2015.

[134] J. A. Duersch and M. Gu, “True BLAS-3 per-formance QRCP using random sampling,”arXiv preprint arXiv:1509.06820, 2015.

[135] I. Guyon and A. Elisseeff, “An introductionto variable and feature selection,” Journalof machine learning research, vol. 3, no. Mar,pp. 1157–1182, 2003.

[136] J. Wright, A. Yang, A. Ganesh, S. Sastry, andY. Ma, “Robust face recognition via sparserepresentation,” IEEE Transactions on Pat-tern Analysis and Machine Intelligence (PAMI),vol. 31, no. 2, pp. 210–227, 2009.

[137] I. Bright, G. Lin, and J. N. Kutz, “Compres-sive sensing and machine learning strategiesfor characterizing the flow around a cylin-der with limited pressure measurements,”Physics of Fluids, vol. 25, no. 127102, pp. 1–15,2013.

[138] S. Sargsyan, S. L. Brunton, and J. N. Kutz,“Nonlinear model reduction for dynamicalsystems using sparse sensor locations fromlearned libraries,” Physical Review E, vol. 92,

33

no. 033304, 2015.

[139] B. Ribeiro and D. Towsley, “Estimatingand sampling graphs with multidimensionalrandom walks,” in Proceedings of the 10thACM SIGCOMM Conference on Internet Mea-surement, IMC ’10, (New York, NY, USA),pp. 390–403, ACM, 2010.

[140] P. Di Lorenzo, S. Barbarossa, P. Banelli, andS. Sardellitti, “Adaptive least mean squaresestimation of graph signals,” IEEE Transac-tions on Signal and Information Processing overNetworks, vol. 2, no. 4, pp. 555–568, 2016.

[141] S. Chen, R. Varma, A. Singh, and J. Kovace-vic, “Signal recovery on graphs: Fundamen-tal limits of sampling strategies,” IEEE Trans-actions on Signal and Information Processingover Networks, vol. 2, no. 4, pp. 539–554, 2016.

[142] S. P. Chepuri and G. Leus, “Subsampling forgraph power spectrum estimation,” in Sen-sor Array and Multichannel Signal ProcessingWorkshop (SAM), 2016 IEEE, pp. 1–5, IEEE,2016.

[143] P. Benner, S. Gugercin, and K. Willcox, “Asurvey of projection-based model reductionmethods for parametric dynamical systems,”SIAM review, vol. 57, no. 4, pp. 483–531, 2015.

[144] K. Taira and T. Colonius, “The immersedboundary method: a projection approach.,”Journal of Computational Physics, vol. 225,no. 2, pp. 2118–2137, 2007.

[145] T. Colonius and K. Taira, “A fast immersedboundary method using a nullspace ap-proach and multi-domain far-field boundaryconditions,” Computer Methods in Applied Me-chanics and Engineering, vol. 197, pp. 2131–2146, 2008.

[146] B. R. Noack, K. Afanasiev, M. Morzynski,G. Tadmor, and F. Thiele, “A hierarchy oflow-dimensional models for the transientand post-transient cylinder wake,” Journal ofFluid Mechanics, vol. 497, pp. 335–363, 2003.

[147] R. Zimmermann and K. Willcox, “An acceler-ated greedy missing point estimation proce-

dure,” SIAM Journal on Scientific Computing,vol. 38, no. 5, pp. A2827–A2850, 2016.

[148] “NOAA optimal interpolation (OI) sea sur-face temperature (SST) v2.” https://www.esrl.noaa.gov/psd/, July 2017.

[149] S. Chaturantabut and D. C. Sorensen, “Astate space error estimate for POD-DEIMnonlinear model reduction,” SIAM Journal onnumerical analysis, vol. 50, no. 1, pp. 46–63,2012.

34

AcknowledgmentsThe authors thank Eurika Kaiser, Joshua Proc-

tor, Serkan Gugercin, Bernd Noack, Tom Hogan,Joel Tropp, and Aleksandr Aravkin for valuablediscussions. SLB and JNK acknowledge supportfrom the Defense Advanced Research ProjectsAgency (DARPA HR0011-16-C-0016). SLB and KMacknowledge support from the Boeing Corpora-tion (SSOW-BRT-W0714-0004). BWB and SLB ac-knowledge support from the Air Force ResearchLabs (FA8651-16-1-0003). SLB acknowledges sup-port from the Air Force Office of Scientific Re-search (AFOSR FA9550-16-1-0650). JNK acknowl-edges support from the Air Force Office of Scien-tific Research (AFOSR FA9550-15-1-0385).