There are two standard frameworks for describing optimal least squares estimation of a random quantity from corrupted measurements. The first technique, Bayesian Least Squares (BLS) estimation, uses explicit models of both the corruption process and the prior distribution of the quantity to be estimated in order to formulate an optimal estimator via Bayes' rule. The second technique, Least Squares regression, uses supervised training on a data set which has clean samples paired with corrupted versions of those samples, to choose an optimal estimator from some family. In many applications, however, one has available neither a model of the prior distribution, nor uncorrupted measurements of the variable being estimated. We will describe a framework for expressing the BLS estimator (regression function) entirely in terms of a model of the corruption process and the density of the corrupted measurements. We show a practical implementation of this nonparametric estimator for additive white gaussian noise (AWGN), and demonstrate the use of this procedure for denoising photographic images, showing that it compares favorably with previously published methods which use explicit prior models. We also describe a dual, prior-free formulation of the Mean Square Error (MSE) which generalizes Stein's Unbiased Risk estimator (SURE), and show how this may be used for unsupervised regression. We then demonstrate the use of this dual formulation in image denoising. In particular, we use the dual formulation to prove the empirically observed fact that, despite their suboptimality, marginal image denoisers chosen to minimize MSE within the subbands of a redundant multi-scale decomposition will always perform better than on the orthonormal versions of those bases. We also develop an extension of SURE that allows minimization of the image-domain MSE for estimators that operate on subbands of a redundant decomposition, and show that this gives improvement over methods which optimize MSE within subbands.