stat.ME — Pith

Top Pith

4

math.FA 2026-05-18 2 theorems

Residual collapse equates ordered POVM realizations by surviving effects

by James Tian

Different orderings and couplings reduce to the same canonical form whose non-escape coordinates are orthogonal and sum to the identity.

abstract click to expand

Ordered realizations of discrete POVMs are studied through a residual transform generated by sequential tests. One application of the transform replaces each coordinate by the effect obtained after all earlier tests have failed, and appends the remaining mass as a terminal outcome. Under natural hypotheses, iterating the transform produces a collapsed POVM whose non-escape coordinates are the parts of the original effects that survive all earlier tests. The resulting collapse map gives an equivalence relation on ordered POVM realizations. Its range and fibers are characterized. The range consists of collapsed POVMs, whose non-escape coordinates are mutually orthogonal and whose support projections strongly sum to the identity. The fiber over a collapsed POVM consists of all ordered realizations with the same residually visible compressions. In particular, different ordered realizations, including ones with different off-diagonal coupling data, can have the same collapsed image. After collapse, the non-escape coordinates are fixed under further residual iteration. The remaining dynamics takes place in the escape effect, which is fragmented by a universal scalar functional calculus.

0

stat.AP 2026-07-03

CESM low-energy matches contrast at 0.874 patient AUC

by Sara Antonijevic, Brani Vidakovic

Masked complex non-decimated wavelet features for patient-level classification of contrast-enhanced mammography

Leakage-free wavelet classifier shows the two image types are equivalent yet use separate phase and magnitude channels.

abstract click to expand

Contrast-enhanced spectral mammography (CESM) acquires two images of each breast, a low-energy image and a recombined contrast image, but two questions central to building a classifier on them remain unsettled: whether the two image types carry comparable malignancy signal, and how a patient's several images should be combined into a single decision. Both are hard to answer reliably, because most published CESM classifiers split cross-validation folds at the image level, letting images of the same patient fall in both training and test sets and inflating reported performance. We pair a masked complex non-decimated wavelet feature bank with an elastic-net logistic classifier, evaluated under repeated patient-grouped nested cross-validation with patient-cluster bootstrap inference on the CDD-CESM dataset (1,880 images, 308 patients); under this leakage-free evaluation the inflation from testing on previously seen patients is negligible. On normal-versus-malignant detection, the two acquisitions are statistically indistinguishable in patient-level AUC under the proposed evaluation framework. Under single-image fusion the contrast image reaches a patient-level AUC of 0.874 (95% CI 0.827-0.918) and the low-energy image is statistically indistinguishable from it, yet the two encode malignancy through disjoint, interpretable channels: phase coherence on the low-energy image and magnitude distribution on the contrast image. The framework matches a pretrained ResNet-50 representation at the patient level, but whereas the frozen deep representation is not directly interpretable at the level of individual predictors, every predictor in the wavelet representation carries an explicit physical meaning. The result is a transparent, leakage-free baseline against which future CESM classifiers can be measured.

0

stat.ME 2026-07-03

The paper develops design-based inference for experiments that randomly assign…

by Jiawei Fu, Cyrus Samii +1 more

Inference for Group Interaction Experiments

In a sparse-sampling regime, standard cluster methods account for dependencies from interference and group formation.

abstract click to expand

A common experimental research design is one in which individuals are randomly allocated into groups that then interact under different group-level treatment conditions. We develop design-based inference for such "group interaction" experiments, covering scenarios in which groups are either fixed or randomly formed and in which potential outcomes are either fixed relative to others' group assignments or subject to interference. For each scenario, we characterize the causal estimand that the design targets and the inferential strategy appropriate to it. Working in a sparse-sampling asymptotic regime, we show that cluster-robust inference remains consistent and accounts for dependencies from various sources when interference is present, delivering valid inference on marginalized exposure effects. When interference is absent and groups are formed randomly, the design reduces to an individually randomized experiment, and individual-level heteroskedasticity-robust inference suffices for the average treatment effect. Our results on the asymptotic distribution of commonly used estimators rely on a novel coupling strategy that may be useful for design-based inference in other complex experiments.

0

stat.ME 2026-07-03

Bayesian and quasi-Bayesian estimates merge for Poisson decisions

by Stefano Favaro, Sandra Fortini

Merging of Bayes and quasi-Bayes empirical Bayes procedures for Poisson compound decisions

Concentration rates of marginal PMFs produce matching regret decay, so the faster quasi-Bayesian method performs equivalently in the multidi

abstract click to expand

The Poisson compound decision problem is a long-standing problem in statistics, in which empirical Bayes methods are used to estimate Poisson means under a mixture model. We study this problem from the viewpoint of $g$-modeling, comparing two nonparametric strategies for estimating the unknown mixing distribution: a Bayesian empirical Bayes strategy, based on the Dirichlet process posterior, and a quasi-Bayesian empirical Bayes strategy, based on Newton's algorithm. The latter is computationally attractive, but its relationship with the Bayesian strategy requires theoretical justification. Under a Poisson mixture model with a ``true'', or oracle, mixing distribution, we establish concentration rates for the marginal probability mass functions induced by the Bayesian and quasi-Bayesian estimates. These rates are then translated into rates of decay for the corresponding regrets, interpreted as excess Bayes risks, and used to prove a frequentist merging result between the Bayesian and quasi-Bayesian empirical Bayes strategies. We also extend the analysis to the multidimensional Poisson compound decision problem. Numerical experiments on synthetic data illustrate that the quasi-Bayesian strategy achieves accuracy comparable to the Bayesian strategy, while requiring substantially fewer computational resources, especially in the multidimensional setting.

0

stat.ME 2026-07-03

CAP matches empirical risk at first order and removes second-order bias

by Yijian Huang

Cross-Audit Projection for Model Risk Prediction

Resampling audit plus asymptotic projection corrects over-optimism in binary classification risk estimates without sacrificing leading accur

abstract click to expand

For training-data-based model risk prediction, $K$-fold cross-validation~(CV) is widely used to mitigate the well-known over-optimism of the empirical risk and is often regarded as reliable. However, for binary classification via empirical risk minimization, our numerical studies reveal a surprising phenomenon: $K$-fold CV may perform poorly in estimating class-specific risks, even worse than the empirical estimator. We perform a higher-order asymptotic analysis showing that $K$-fold CV may converge at a slower rate, whereas the empirical estimator exhibits a second-order asymptotic bias that explains its over-optimism. These findings motivate a novel two-step procedure for model risk prediction, termed cross-audit projection (CAP). The cross-audit step adopts the same resampling scheme as $K$-fold CV to estimate over-optimism in subsamples, while the asymptotic-theory-informed projection step adjusts for the reduced sample size in bias correction of the empirical risk. The resulting CAP estimator is first-order asymptotically equivalent to the empirical risk while achieving second-order asymptotic unbiasedness. An accompanying inference procedure is also developed. Simulation studies support theoretical advantages of CAP and demonstrate favorable finite-sample performance. An application to breast cancer detection further illustrates the proposed method.

0

stat.AP 2026-07-03

EVPI extensions show global validation of ADNEX model is complete

by Laure Wynants, Kim Zhipei Wang +6 more

Value-of-Information Analysis for External Validation of Risk Prediction Models in Multicenter Studies and Systematic Reviews

Accounting for center differences reveals when local adoption decisions still need more data to confirm net benefit.

abstract click to expand

External validation studies have finite sample sizes, creating uncertainty about whether a prediction model's Net Benefit (NB) exceeds default strategies' NB. The expected value of perfect information (EVPI) quantifies consequences of uncertainty. Current EVPI methods focus on single studies, ignoring between-center heterogeneity. We extend EVPI and expected value of partial perfect information (EVPPI) to account for between-cluster heterogeneity in multicenter studies and meta-analyses. We distinguish between the global and local optimal strategy and between observed and unobserved clusters. We define EVPIglobal, EVPIcluster_j, EVPIcluster, and EVPPIcluster,prevalence, implemented in the MetaNB R package, and illustrate them using a systematic review across 36 centers of the ADNEX model for ovarian cancer diagnosis. Assuming one global decision regarding ADNEX adoption, there is no need for further data to confirm ADNEX is superior overall (EVPIglobal 0). Meta-analysis borrows information across observed clusters, resulting in consistent local superiority of ADNEX and nonzero but typically lower EVPIcluster_j than when considering local data alone. There is 0.03 probability default strategies are superior in unobserved centers. Eliminating uncertainty on performance and prevalence in each (EVPIcluster) would gain 1134 net avoided false positives (FP) per year, assuming 350000 tumors annually with 20% malignancies. Determining only local prevalence with certainty (EVPPIcluster, prevalence) would gain net 158 avoided FP per year. EVPI extensions disentangle sources of uncertainty and quantify the need for further validation to determine the global or locally optimal strategy. Considering uncertainty and heterogeneity in clinical utility across clusters is essential to decide whether additional validation studies are warranted.

0

stat.ME 2026-07-03

Multipliers create nondegenerate tests for Fréchet regression

by Leheng Cai, Xu Guo +1 more

MATCH: Multiplier-Assisted Tests for Conditional Hypotheses in Non-Euclidean Data

Sample splitting and random multipliers on held-out losses produce Gaussian limits without tangent coordinates or residual terms.

abstract click to expand

We propose a new procedure MATCH (Multiplier-Assisted Tests for Conditional Hypotheses) to test whether the non-Euclidean data match the target model, which is a general framework for significance and specification testing in Fr\'echet regression. MATCH covers global significance, partial significance, and the adequacy of global Fr\'echet regression, providing a unified way to compare unrestricted conditional Fr\'echet means with restricted alternatives. One of the key challenges is that the ordinary held-out loss difference is first-order degenerate under the null: the oracle losses coincide, and plug-in statistics is dominated by nuisance estimation error. MATCH uses sample splitting and independent random multipliers on held-out losses to create a nondegenerate Gaussian leading term without residuals or tangent-space coordinates. To improve data use and stability, we further develop cross-fitted tests and repeated cross-fitting with p-value merging. We establish asymptotic null validity, consistency under fixed alternatives, and local power guarantees. Simulations for distributional, symmetric positive-definite (SPD) matrix-valued, and spherical responses support the theoretical findings, and applications to county-level household income distributions and North Atlantic tropical-cyclone locations demonstrate the practical use of the proposed tests.

0

stat.ME 2026-07-03

Joint distributions of sample statistics power new GOF tests

by Roman Guchenko

Goodness of Fit Tests Based on Joint Densities of Multiple Sample Statistics

Simulations show the procedures match or exceed classical and Zhang methods for continuous nulls with known parameters.

abstract click to expand

We propose goodness-of-fit tests based on simulated confidence sets for joint distributions of multiple sample statistics, focusing on absolutely continuous null distributions with known parameters. One class of tests uses hyperrectangular confidence sets for principal components of order statistics and related statistic vectors. Extending earlier work on horizontal and vertical confidence bands for cumulative distribution functions, these tests are compared with some classical, Zhang, and related graphical tests. Simulations show that the proposed procedures are competitive with, and often more powerful than, existing methods. We also study the geometry of principal-component-based statistics; under a normal null distribution, the first principal component corresponds to the sample mean, while the second is related to a linear analogue of variance. A second class of tests uses confidence sets of arbitrary shape constructed through highest density regions. Unlike earlier kernel-density-based approaches, we use a k-nearest-neighbor method for detecting highest density regions, which is better suited to higher-dimensional statistic vectors. We study tests based on order statistics, empirical distribution function values, moments, and combinations of classical goodness-of-fit statistics. The resulting procedures are powerful against a wide range of alternatives. We also outline a two-sample extension via permutation tests based on joint distributions of several statistics and compare moment-based versions with energy-distance permutation tests. Finally, we discuss transformations other than the probability integral transform, showing that mapping data to another target distribution, such as the standard normal, can be advantageous when powerful tests are available for that distribution.

0

stat.ME 2026-07-03

Weighted tilt restores coverage for censored label shift

by Seungjin Choi

Conformal Bayes for Two-Sided Censored Gaussian Regression under Label Shift

Mixed atom-density calibration weights yield smaller valid sets than source-score methods in two-sided censored Gaussian regression.

abstract click to expand

Prediction under label shift becomes nonstandard when responses are censored. In a two-sided censored Gaussian model, latent values below $L$ and above $U$ are recorded at the boundary values, so the observed predictive distribution is mixed, with atoms at $L$ and $U$ and a continuous density on $(L,U)$. In this paper we develop conformal Bayes for this mixed-space setting by combining posterior predictive tilting with weighted conformal calibration. Under a two-sided Tobit Gaussian Bayesian prediction head with a Laplace posterior approximation, the tilted predictive distribution has left-atom, interior, and right-atom components, with a three-term closed-form normalizer. The resulting prediction set is a mixed highest density region that can combine boundary atoms with an interior interval and can reduce to atom-only sets under strong censoring. The main technical issue is that latent label shift does not directly give an ordinary density ratio on the observed censored scale. A latent exponential tilt induces tail-averaged atom weights at the censored boundaries, while the interior ratio remains density based. This yields a mixed observed-space calibration weight with two atom ratios and one interior density ratio. The weight corrects the calibration measure, while predictive tilting gives target-adapted mixed-HDR geometry. Synthetic experiments show that weighted tilted conformal Bayes restores marginal coverage with smaller sets than weighted source-score calibration, while revealing a trade-off between marginal coverage and component-wise behavior across atoms and interior observations.

0

stat.ME 2026-07-03

Expert portfolio detects PDE model errors missed by residuals

by Ieva Kazlauskaite

Sequential Structure-Sensitive Residual Diagnostics for PDE Inverse Problems

Sequential e-process rejects bad fits early using spatial residual patterns with anytime-valid error control.

abstract click to expand

Computational models in science and engineering are often assessed by checking whether the residual norm is consistent with the assumed noise level. This can be misleading in smoothing inverse problems: structured model errors may be attenuated in observation space, leaving residual magnitudes below practitioner discrepancy thresholds while coherent residual patterns remain. As a result, residual-norm diagnostics can accept fitted models that still give biased parameters, predictions, or quantities of interest. We propose a structure-sensitive sequential diagnostic based on e-processes. The method uses a portfolio of spatial residual-pattern experts, updates their likelihood-ratio wealth as observations are processed, and rejects the fitted model when the aggregate wealth crosses a prescribed threshold, giving anytime-valid type-I error control for a fixed fitted model. We compare the method with Morozov discrepancy checks, fixed-sample residual tests, and batch projection tests. Across three inverse problems (elliptic diffusion, two-dimensional Stokes flow, and a glaciological ice-stream inversion implemented in the community finite-element model icepack) we demonstrate how standard discrepancy checks accept misspecified fits that produce materially wrong quantities of interest. Structure-sensitive batch tests detect these failures using the full dataset, while the e-process detects them earlier from a fraction of the observations. After rejection, the expert wealth attributes the evidence to residual patterns in the chosen dictionary and provides a basis for exploratory model correction.

0

econ.EM 2026-07-03

GIV consistent at √T only when few units dominate aggregate

by Gokul Gopalan Ramachandran

Granular Instrumental Variables in Large Panels: Identification and Inference Across Strong, Nearly Weak, and Weak GIV

Three regimes of instrument strength arise from unit sizes in growing panels, dictating rates and valid inference methods.

abstract click to expand

I develop the asymptotic theory of instrument strength for Granular Instrumental Variables (GIV) in large panels with both $N$ and $T$ growing. The strength of the GIV depends on the presence of dominant units. I formalise what dominance means and characterise three regimes of instrument strength. When a few units dominate the aggregate, the instrument is strong. The GIV estimator is consistent and asymptotically normal at the standard $\sqrt{T}$ rate. When large units stand out but do not dominate, the instrument weakens. But I show that the parameter of interest remains recoverable. The GIV estimator remains consistent and asymptotically normal, now at a rate slower than $\sqrt{T}$. When units are comparable in size and none stands out, the instrument is weak in the standard sense. The GIV estimator is inconsistent and has a non-standard distribution. Wald inference is reliable only outside the weak regime. When the instrument is weak, I recommend Anderson-Rubin confidence sets. In practice, the instrument must be constructed in a first stage. I show that the feasible estimator attains the same rate, but its asymptotic variance picks up an additional term from the first-stage estimation. Valid inference must use standard errors that account for this term. I apply the GIV estimator with the correct standard errors to recover the short-run demand elasticities of three commodities: refined copper, crude oil, and natural gas.

0

stat.ME 2026-07-03

Orthogonal arrays and difference schemes create larger grouped arrays

by Meixin Liu, Chunyan Wang +2 more

Grouped Orthogonal Arrays from Orthogonal Arrays and Difference Schemes

Provides new designs with more groups and larger sizes for experiments assuming negligible cross-group interactions.

abstract click to expand

Grouped orthogonal arrays were introduced to address experimental design problems arising in computer experiments with grouped inputs, as well as in physical experiments where interactions between factors from different groups are assumed to be negligible. Motivated by the growing need for flexible and efficient designs under such settings, this article develops several constructions to expand the existing catalogs of grouped orthogonal arrays. The proposed constructions provide a large collection of new grouped orthogonal arrays with significantly larger numbers of groups and group sizes.

0

stat.ME 2026-07-03

Plausibility enables exact inference in general parametric families

by Stefan Böhringer, Jesse Swen

Plausibility: Exact inference in R

R package implements the framework for regression models and supports exact tests on data examples.

abstract click to expand

Plausbility is a theoretical framework that allows to conduct exact inference in general parametric families. We introduce R-packages {\em plausibility} that implements this framework for a wide class of regression models. Plausibility can also be used to test penalized regression models such as estimated by package {\em glmnet}. We illustrate the package using a number of R data sets Through a class-based mechanism, the package can be easily extended. We illustrate and discuss computation aspects of the implementation and their impact on real-data analysis.

0

stat.ME 2026-07-03

Moment method selects random effects consistently in mixed models

by Yifan Chen, Yuedong Wang +1 more

Moment-Based Selection of Multiresponse Linear Mixed-Effects Models

It reduces the problem to convex optimization using cross-moment identities and establishes finite-sample guarantees under sub-Weibull error

abstract click to expand

We propose MOMENT (\textbf{MO}ment-Based \textbf{M}ixed-\textbf{E}ffects Selectio\textbf{N} and Es\textbf{T}imation), a stage-wise moment-based framework that exploits second-order cross-moment identities to select and estimate the random-effects covariance matrix and fixed-effects coefficients. By inducing sparsity through its diagonal under a positive semidefinite constraint, the random-effects selection problem reduces to a smooth constrained convex optimization problem that can be solved efficiently by projected gradient descent. We further establish finite-sample theoretical guarantees for the proposed procedure, including random-effects selection consistency and fixed-effects selection consistency under joint sub-Weibull errors. Simulation studies show that MOMENT performs competitively overall and can substantially outperform separate univariate analyses when responses are correlated. An application to the hemodialysis dataset demonstrates that the proposed method yields an interpretable and flexible approach for multivariate longitudinal data.

0

stat.AP 2026-07-03

IRT model extracts rider skill and condition difficulty from binary outcomes

by Fabio Carucci

Inverse Suitability: Identifying Condition Difficulty and Rider Skill from Behavioural Outcomes via Continuous-Item Response Theory

Continuous-item formulation recovers skill at r=0.96 and improves Brier score by 0.33 over expert curves on synthetic cohort of 80 riders.

abstract click to expand

Suitability scoring for outdoor activities (kitesurfing, paragliding, ski touring) maps environmental conditions to a go/no-go verdict via expert-defined curves. These curves conflate two distinct quantities: the intrinsic difficulty of a condition and the skill of the person facing it. We introduce Inverse Suitability, a continuous-item Item Response Theory (IRT) model that identifies both from behavioural outcomes alone. Each outcome is a triple (rider r, condition metric x at site s, binary outcome y); we model P(y=1) = sigma(a (theta_r - delta(x, s))), where theta_r is latent rider skill, delta(x, s) is a latent difficulty function anchored to a physics-derived expert curve as its prior, and a is a discrimination parameter. The formulation is strictly more general than a single suitability curve, which it recovers exactly when skill is integrated out under the population distribution. Parameters are estimated by marginal maximum likelihood with Gauss-Hermite quadrature; identification holds when the rider-by-condition incidence graph is connected, with a documented single-curve fallback otherwise. We validate via synthetic recovery: on a reference cohort (80 riders times 30 outcomes) the model recovers latent skill at r = 0.96, locates the difficulty minimum within 3 units of ground truth, and improves held-out Brier Skill Score by +0.33 over the expert-curve baseline. The recovered difficulty function defines a measurable, site-level construct, an intrinsic difficulty atlas, that existing meteorological observation networks do not capture. All results reproduce from a single command on synthetic data, requiring no proprietary observations.

0

stat.ML 2026-07-03

Autorelevance function recovers lag structure in time series forecasts

by Julian Cardenas, Jamie Arjona +1 more

Autorelevance function and other feature relevance measures for univariate time series

Shapley-based measures with one-step forecast replacement for missing lags identify expected patterns across ARMA and neural models.

abstract click to expand

We propose a model agnostic methodology to measure lag relevance in machine learning forecasting models applied to univariate time series. Particularly, we are working in the context of time series using the frameworks of Ghost variables and Shapley values, together with additive importance measures, to introduce the auto-relevance and partial auto-relevance functions as the lag importance values. Additionally, we propose a novel method to replace absent features in coalition based methods with a one step forecast from the same model. We evaluate these proposals under different simulations and real data cases. This combined framework perspective is particularly suitable for time series. In addition, to show our discoveries we use a pull of models from the seasonal ARMA family and recurrent neural networks. We found that the calculated relevance measures successfully demonstrate the expected lag structure in almost all cases.

0

math.ST 2026-07-03

Perturbation theory transfers sup-norm rates to functional principal components

by Hajo Holzmann, Kevin Wilk

Transferring supremum-norm rates and weak convergence of covariance kernel estimators to functional principal components

L2-perturbation theory converts existing covariance kernel rates into optimal sup-norm and normality results for the associated eigenfunctio

abstract click to expand

We show that $L_2$-perturbation theory can be used to transfer rates of convergence in the supremum norm as well as weak convergence in the space of continuous functions from covariance kernel estimators to the associated functional principle components (FPCs). As an application we obtain optimal rates of convergence in sup-norm, including minimax-lower bounds, as well as asymptotic normality for estimating the FPCs in a discrete observational model with errors under fixed, synchronous design. The sparse to dense transition which has previously been observed for mean function and covariance kernel estimators also applies to the FPCs. Surprisingly, eigenvalue estimation exhibits a discretization-dominated regime under sparse designs, too. Our results further apply to estimators of cross-covariance and long-run covariance kernels, as well as to covariance kernels of derivative processes. We also present results of numerical experiments in which we use the Nystr\"om method to compute FPCs and eigenvalues, and give an empirical illustration to series of daily temperature curves.

0

stat.ME 2026-07-03

Closed-form wrapped Gaussians replace Laplace for posterior approximation

by Marcelo Hartmann, Luu Hoang Phuc Hau +6 more

Beyond Laplace: Closed-form wrapped Gaussian posterior approximations on statistical manifolds

Contrast functions approximate maps on statistical manifolds, removing geodesic solvers and curvature calculations.

abstract click to expand

In Bayesian statistics, the Laplace approximation provides a computationally efficient approximation to posterior distributions. However, its Gaussian form restricts it to elliptical shapes, limiting its ability to capture important posterior features such as skewness, heavy tails, and narrow high-probability regions. Recent work has addressed this limitation by exploiting Riemannian geometry to push forward Gaussian distributions from the tangent space to the manifold, referred to wrapped Gaussians. While offering greater flexibility, they introduce substantial computational challenges. Sampling requires solving geodesic equations through the exponential map and density evaluation additionally depends on the logarithmic map and Jacobi fields, involving costly differential equation solvers and geometric quantities such as inverse matrices, Christoffel symbols and curvature tensors. To overcome these limitations, we employ the theory of contrast functions to derive tractable approximations of the logarithmic and exponential maps on statistical manifolds endowed with the Fisher--Rao metric and the prior distribution geometry. The resulting methodology bypass the need to compute these geometric quantities and numerical solvers thereby removing the principal computational bottlenecks of existing wrapped Gaussian approaches. Empirical results across a range of models demonstrate that the proposed approximation captures complex posterior geometries while remaining orders of magnitude faster than current state-of-the-art approximation.

0

stat.ME 2026-07-03

Calibration lets multimodal predictors borrow across missingness patterns

by Junhan Yu, Kejian Zhang +2 more

Pattern-Calibrated Multimodal Prediction under Blockwise Missingness

Bounds decompose error into overlap size, calibration gap, and representation error, showing when borrowing beats local fitting.

abstract click to expand

Blockwise missingness in multimodal data is usually treated as an incomplete-input problem. We instead focus on prediction for a prespecified observed-modality pattern, where the observed modality set determines the information on which the prediction rule can condition. A procedure that imputes missing modalities, zero-fills unobserved modalities, or trains a single pooled predictor may borrow information across patterns, but it can also mix pattern-specific prediction rules. We propose Multimodal Overlap-aware Shared-specific Alignment and Inter-pattern Calibration (MOSAIC), a pattern-calibrated framework for borrowing across missingness patterns without collapsing their prediction rules. MOSAIC learns shared and modality-specific representations, uses the available representations that overlap with the target pattern to fit a first-stage predictor, and then estimates the calibration gap from target-pattern data. We establish non-asymptotic bounds that decompose the error into overlap effective sample size, calibration gap, and representation-learning error, clarifying when cross-pattern borrowing improves over local fitting and when the improvement is controlled by rule mismatch or representation-learning error. Simulations examine representation recovery and target-pattern correction, and applications to ICU mortality prediction, emotion recognition, and glaucoma classification show gains when target-pattern samples are limited or pattern-specific rules differ.

0

stat.ME 2026-07-03

Transportability estimates effect changes from modifier prevalence shifts

by Michael Cheung, Candus Shi +4 more

From Subgroups to Population Composition: A Transportability Approach to Effect Heterogeneity

Modeling effects in hypothetical populations with shifted prevalences ranks characteristics by their link to differential vulnerability.

abstract click to expand

Identifying heterogeneous populations across which exposure effects vary is essential for transportability applications, cost-benefit analyses, and intervention prioritization. Traditional methods for heterogeneity analyses rely on parametric regression with prespecified subgroups, which may fail to capture complex patterns of effect modification. While recent data-adaptive methods improve high-dimensional heterogeneous effect prediction, they add methodological complexity to analyses and may offer limited insight into key drivers of heterogeneity. In this paper, we propose a novel, conceptual approach for heterogeneity analyses that considers how exposure effects would differ in populations with different compositions by modeling the population-level effect surface as a function of the distribution of effect modifiers. The approach consists of three steps: i) selecting confounders and effect modifiers based on prior knowledge (or alternatively using data-adaptive methods to learn effect modifiers), ii) estimating exposure effects in hypothetical populations with different effect modifier prevalences using transportability methods, and iii) modeling the estimated effects as a function of prevalence values. This approach provides two types of outputs: estimation of the change in the population-level exposure effects attributable to increases in effect modifier prevalence and ranking of effect estimates across multiple effect modifiers and prevalences to identify population characteristics most strongly associated with differential vulnerability. We demonstrate the approach using Demographic and Health Surveys data to examine heterogeneous effects of drought on child stunting and provide a Shiny application to implement this approach in any setting.

0

stat.ME 2026-07-03

Lancaster copulas arise from orthogonal expansions of Lancaster probabilities

by Angelo Efoevi Koudou, Yves I. Ngounou Bakam +1 more

Lancaster copulas

The construction supplies infinite series for the copula and density whose low-order truncations already match target dependence in numerica

abstract click to expand

We introduce a new copula class, called Lancaster copulas, built from orthogonal expansions of continuous Lancaster probabilities. We derive infinite-series representations for the copula and its density, study truncation effects, and show in numerical experiments that low-order truncations already provide accurate approximation.

0

stat.ME 2026-07-03

Spike-and-slab prior recovers multimorbidity clusters in EHR data

by Oyebayo R. Olaniran, Soumya S. Paria +3 more

Continuous-Time Bayesian Networks with Structured Shrinkage Priors for Modelling Multimorbidity Trajectories in Large-Scale Electronic Health Records

Structured CTBN model on 33,558 UK Biobank participants identifies cardiometabolic and inflammatory disease modules.

abstract click to expand

Multiple long-term conditions (MLTCs) arise through complex, time-dependent interactions among diseases, yet existing methods often struggle to jointly model disease progression, multimorbidity networks, and high-dimensional risk factors. We propose a structured Bayesian continuous-time Bayesian network (CTBN) framework for learning directed disease-dependency networks from longitudinal electronic health records. The model allows disease transition intensities to depend on existing conditions, pairwise disease interactions, and exogenous covariates. To control the combinatorial growth of interaction parameters, we introduce order-dependent shrinkage priors that increasingly penalise higher-order effects while preserving clinically interpretable main effects. We compare four sparsity-inducing priors, spike-and-slab, structured normal, Bayesian LASSO, and regularised horseshoe through extensive simulation studies. Across multiple data-generating scenarios, the spike-and-slab prior achieved the best network recovery, variable-selection accuracy, and false-discovery control, while continuous shrinkage priors were less effective for hard variable selection. The proposed framework was applied to UK Biobank primary care records, focusing on data from 33,558 participants who were free of the ten selected most prevalent conditions at age 40 and who subsequently developed at least one of these conditions during the follow-up period. The selected spike-and-slab model identified two dominant disease modules: a cardiometabolic cluster centred on diabetes and an inflammatory cluster linking respiratory and atopic conditions.

0

cs.AI 2026-07-02

AI agents reproduce 72% of human ideological gaps in data analysis

by Jiacheng Miao, Jonathan K Pritchard +1 more

The Agentic Garden of Forking Paths

Different personas cause agents to reach opposing conclusions from the same dataset, showing selective reporting among valid paths is the co

abstract click to expand

Empirical research rarely admits a unique analysis. Different analytical choices can lead to different conclusions from the same data, yet these hidden forking paths are difficult to observe. We show that AI agents capture much of the analytical variation among human researchers while making these paths explicit. Across four high-stakes domains, assigning different personas is sufficient for AI agents to report divergent, often opposing, conclusions from the same data and question, with findings systematically aligned with those beliefs. In a study in which 42 human research teams analyzed the same immigration dataset, AI agents reproduced 72% of the human ideological gap in reported effect estimates. Despite reaching opposing conclusions, it is difficult to identify clear issues in each analysis based on the final AI reports: 86% passed independent AI review and 78% passed majority human expert review. These findings suggest that the central challenge is often not flawed analyses, but selective exploration and reporting from a large space of methodologically defensible analyses. AI agents may amplify this longstanding problem by making such exploration inexpensive and scalable. To address this, we introduce the m-value (multiverse value), the probability that an analysis path would produce a claim at least as extreme as the reported one. We further introduce Agentic Bootstrap, which estimates the m-value by using AI agents to sample plausible analysis paths. Applied to the human immigration study, 13.5% of reported human analyses fell in the most extreme 5% of the analysis space (m<0.05). Scientific evidence should therefore be evaluated not only by a single reported analysis but also by its position within the distribution of analyses that could reasonably have been reported. Agentic Bootstrap makes this distribution observable and turns it into a criterion for scientific credibility.

0

stat.AP 2026-07-02

Parameter uncertainty reduces epidemic sensitivity and yields conservative policies

by Nicholas R. Wu, Michael C. Fu

Sensitivity Analysis and Optimization of Stochastic Epidemic Models under Parameter Uncertainty

Unbiased estimators for stochastic models show weaker herd-immunity effects and more cautious intervention levels when parameters are drawn

abstract click to expand

To address sensitivity analysis and optimization for a discrete-time stochastic epidemic model, we derive unbiased gradient estimators that accommodate uncertainties represented as distributions over the parameters of interest, such as those arising from Bayesian calibration. Specifically, we estimate the sensitivity of total infections over a finite time horizon with respect to the proportion immunized ($v$) and the contact rate ($\beta$). Comparing the proposed estimators with deterministic limit approximations based on large populations reveals differences due to the finite population and time horizon. The estimators exhibit lower variance than finite-difference estimators for the derivative with respect to $\beta$, but higher variance for the derivative with respect to $v$. Simulation experiments indicate parameter uncertainty reduces sensitivity to the parameters of interest. In particular, indirect effects of vaccination, such as herd immunity, are less pronounced compared to when parameters are known. For optimization problems balancing intervention and infection costs, incorporating parametric uncertainty leads to more conservative policies.

0

stat.AP 2026-07-02

20-50 sampled points per region lift spatial scan power

by Foad Namjoo, Drew McClelland +2 more

Sampling for Region-Aggregated Spatial Scan Statistics

Uniform sampling from each area's geometry and even value spreading recovers most detection ability lost by using centroids alone.

abstract click to expand

Anomaly detection in geospatial data is a crucial tool in geographic information science (GIS), with applications ranging from national security to public-health surveillance to the study of societal disparities. This work focuses on spatial scan statistics and addresses a key mismatch: spatial counts are typically aggregated into predefined regions (census tracts, zip codes, counties), whereas the most efficient scan algorithms operate on spatial point data. The standard remedy -- collapsing each region to its centroid, as in widely used tools such as SaTScan -- is convenient but, as we show, discards the region's spatial extent and causes a significant loss in statistical power. To resolve this, we propose a simple yet scalable fix: replace each spatial region with 20-50 points sampled uniformly from its geometry and spread the region's values evenly across them. This approach improves statistical power while maintaining computational tractability. A convergence analysis explains why so few samples per region suffice. We recommend this sampling-based conversion as the default way to apply point-based spatial scan statistics to region-aggregated data for anomaly detection.

0

stat.ME 2026-07-02

Score test distinguishes non-nested cure-fraction survival models

by Cynthia A. V. Tojeiro, Francisco Cribari-Neto +2 more

J- and MJ-Type Tests for Non-Nested Parametric Survival Models with a Cure Fraction: A Score Test Approach

The MJ statistic checks whether at least one candidate model is correct and supplies a selection rule, all from null-model estimates alone.

abstract click to expand

We propose specification tests for discriminating among non-nested parametric survival models with a cure fraction, focusing on models that differ only in their baseline distributions. The proposed approach augments the null log-likelihood with information from competing models and applies a score test to assess whether the additional information is redundant. Because the test relies only on restricted maximum likelihood estimates, it avoids fitting augmented models. For two competing models, the score statistic reduces to a quadratic form in the sample mean of the individual log-likelihood differences. We show that its signed square root coincides with Vuong's test statistic, although our framework differs in three important respects: it tests the specific null hypothesis that a given model is the true data-generating process, it uses an unsigned statistic that extends naturally to $M \ge 2$ competing models, and it estimates the Kullback-Leibler bias by parametric bootstrap. The resulting MJ statistic combines the individual J tests to assess the global null hypothesis that at least one candidate model is correctly specified, while also providing a model-selection criterion.

0

stat.ME 2026-07-02

Baseline treatment identifies effects despite informative switching

by Yang Liu, Andrew Ying +2 more

An Instrumental Variable Approach to Account for Informative Treatment Switching in Real-world Evidence

The doubly robust estimator uses an instrumental variable and martingale residuals, without needing a no-switching subset, and applies to mu

abstract click to expand

Reproducible and generalizable assessment of treatment decisions requires principled handling of subsequent treatment switching that may inform expected outcomes and shift across cohorts and over time. To effectively account for informative treatment switching, we propose an instrumental variable approach that characterizes the poorly documented expected outcomes at switching as unmeasured confounding. After establishing the baseline treatment as a viable instrumental variable, we constructed an estimating equation based on the association between the centered instrumental variable and a martingale style residual process that identifies the treatment effect under structural cumulative survival model. Our proposed method is doubly robust, i.e., valid whenever either of baseline propensity model or no-switching outcome model is consistently estimated. A co-training of treatment effect parameter and survival outcome regression model eliminated the requirement of observing a no-switching subset under semi-parametric additive hazards models. We further developed an baseline-survival-corrected cross-fitting approach to incorporate general machine learning models for estimating nuisance models. Numerical results demonstrated the validity of our method in various settings when a basket of benchmark solutions produced biased or contradictory results. We applied our method to comparison of high-efficacy vs standard efficacy disease modifying treatments as the second line therapy of multiple sclerosis.

0

stat.ME 2026-07-02

GGMNIRA quantifies node influence via KL after mean manipulations

by Yiming Wu, Fei Wang +1 more

Simulating Node Manipulations in Gaussian Graphical Models: The GGMNIRA Framework for Continuous and Ordinal Psychological Network Data

The algorithm simulates conditional mean changes to quantify how node manipulations alter psychological network distributions.

abstract click to expand

Scientific Abstract: In psychological network analysis, centrality indices are commonly used to evaluate the importance of nodes within a network. However, centrality only captures the static topological position of a node, and there is no sufficient theoretical justification for assuming that it reflects a node's influence on network dynamics. The NodeIdentifyR Algorithm (NIRA) offers an alternative by systematically applying simulated manipulations to node intercepts within the Ising model to evaluate nodes' projected importance, but this algorithm is restricted to binary data, and the manipulated parameter lacks a clear theoretical meaning outside the context of psychopathology. To address these limitations, we propose the Gaussian Graphical Model NodeIdentifyR Algorithm (GGMNIRA), which manipulates a node's conditional mean and uses Kullback-Leibler (KL) divergence to quantify the change in network distribution before and after manipulation, thereby extending this simulated manipulation logic to the Gaussian graphical model framework, which is applicable to continuous and ordinal data. Around this algorithm, we further developed a correlation stability coefficient and a nonparametric bootstrap difference test for KL divergence, with corresponding interpretive thresholds established through simulation studies. The framework was also extended to bridge Gaussian graphical models and moderated Gaussian graphical models, enabling its application to multi-construct comorbidity networks and to contexts involving moderation effects. All methods are implemented in the R package "GGMNIRA".

0

stat.ME 2026-07-02

Transfer learning yields consistent quantile regression estimators

by Gabriela Ciuperca

Transfert learning and adaptive LASSO quantile

Two L1 penalties from a source database estimator deliver sparsity and faster computation than standard adaptive LASSO.

abstract click to expand

We propose for a quantile regression an estimation method for transferring knowledge using two $L_1$ penalties based on an estimator obtained from a source database. The proposed transfer learning estimator satisfies the properties of consistency and sparsity. Its convergence rate and asymptotic behavior are studied in several scenarios. This knowledge transfer results in a shorter computation time than that of the standard adaptive LASSO estimator. Another advantage of our method is that it can be applied to models with non-Gaussian errors. In addition, in order to implement the computing of the adaptive transfer LASSO quantile estimator, we propose an algorithm. The simulations confirm the theoretical results and demonstrate that the adaptive learning estimator, calculated using the proposed algorithm, is more competitive than the LASSO estimators. Finally, we illustrate the practical utility of the proposed transfer learning estimator and algorithm using a real-data application involving the physicochemical properties of protein tertiary structures.

0

stat.ME 2026-07-02

Latent achievement enters self-efficacy regression via conditional copula

by Sarah Lee, Matias Quiroz +1 more

How does academic performance affect self-efficacy? Interpretable modelling through latent academic achievement

Formulation yields interpretable link and faster variable selection than joint model in mixed-scale data

abstract click to expand

There is increasing evidence of a directional relationship from academic performance to self-efficacy. We develop a Bayesian model for investigating this relationship when academic performance is measured on an ordinal scale and self-efficacy on a continuous scale. The model allows latent academic achievement to enter the self-efficacy regression as a predictor, while Bayesian variable selection identifies factors associated with either response. The resulting conditional formulation yields an interpretable regression characterisation of how latent academic achievement relates to self-efficacy. Furthermore, it enables a tailored partially collapsed Gibbs sampler that analytically integrates out the regression coefficients when updating the variable inclusion indicators. Simulation studies demonstrate that the proposed conditional formulation and tailored sampler improve sampling efficiency and variable-selection performance relative to a recent, more general joint Gaussian copula regression formulation. We apply the methodology to data from the longitudinal study of Australian children, a landmark national cohort study covering children's education, social and emotional wellbeing, health and family circumstances. The model and analysis shed light on how latent academic achievement relates to self-efficacy in Australian children, and reveal that the two outcomes differ markedly in the range of covariates associated with each outcome.

0

stat.ME 2026-07-02

Distributed estimator reaches two-phase minimax rates for unidentifiable prediction

by Erbo Li, Zhaojun Hu +3 more

Distributed Prediction under Heterogeneity with Unidentifiable Parameter

Trace-similarity penalty and invex relaxation deliver model-free bounds with lower communication cost.

abstract click to expand

Predicting a response based on covariates is a fundamental problem in statistics and machine learning. However, profound difficulties arise when the underlying low-dimensional structural parameters are unidentifiable, as typified in dimension reduction contexts. Specifically,estimating these non-identifiable parameters inherently introduces severe nonconvexity. In distributed settings, this difficulty is further compounded by the challenges of data heterogeneity and communication cost. To overcome these intertwined barriers, we propose a novel distributed semiparametric framework. We formulate an adaptive homogeneity pursuit utilizing a trace-similarity penalty to effectively address data heterogeneity. To resolve the ensuing severe nonconvexity and communication bottlenecks, we introduce an invex relaxation technique coupled with a multi-step local update algorithm, ensuring stable convergence to global optimality with significantly reduced communication overhead. Theoretically, we establish a non-asymptotic model-free prediction error bound and prove that our estimator achieves a two-phase minimax optimal convergence rate and an sharper model-free prediction error bound. Furthermore, we provide theoretical guarantees for algorithmic convergence and communication efficiency. Extensive simulations and a real-world multi-center medical application validate the superiority of our method.

0

stat.ME 2026-07-02

Intervals achieve nominal coverage for risk difference in paired data

by Jia Zhou, Chang-Xing Ma

Confidence Intervals for the Risk Difference in Combined Unilateral and Bilateral Data Incorporating a Distribution-Based Approach

The distribution-based method matches existing widths while reflecting skewness that asymptotic approaches miss in small samples.

abstract click to expand

Combined unilateral and bilateral binary outcomes frequently arise in studies involving paired organs. The risk difference is a clinically interpretable measure for comparing treatment effects between groups. Existing confidence interval methods are primarily based on asymptotic normality and may fail to adequately reflect finite-sample distributional features, particularly skewness. To address this issue, we propose a distribution-based confidence interval derived from the probability distribution of the risk difference estimator and a modified MOVER procedure that accounts for intra-subject correlation. Their performances are compared with those of commonly used asymptotic methods through extensive simulation studies. Across a broad range of parameter settings, all methods exhibited satisfactory performance as sample size increased. The proposed distribution-based interval achieved coverage probabilities close to the nominal level with interval widths comparable to those of existing procedures. In small sample settings, it was able to capture skewness in the sampling distribution that was not reflected by methods relying on asymptotic normality. Analyses of two real-world datasets demonstrated the practical applicability of the competing methods and yielded consistent inferential conclusions. The proposed approach provides an alternative framework for interval estimation of the risk difference in studies involving combined unilateral and bilateral binary outcomes.

0

stat.ME 2026-07-02

Selective borrowing lets hybrid trials use external data safely

by Ke Zhu, Hairong Huang +2 more

Robust Estimation and Inference with Selective Borrowing in Hybrid Controlled Trials: A Tutorial with SelectiveIntegrative and intFRT

Tutorial workflow covers eligibility alignment, matching and selective strategies with R packages to improve efficiency while keeping infere

abstract click to expand

Hybrid controlled trials (HCTs) augment randomized controlled trials (RCTs) with external controls (ECs) to improve statistical efficiency when RCTs face limited sample sizes, slow accrual, or ethical constraints. However, valid use of ECs requires careful adjustment for covariate shift and outcome drift, as inappropriate borrowing may introduce bias and compromise inference. This tutorial provides a practical workflow for estimation and inference in HCTs. We first present a statistical analysis roadmap covering estimands, identification assumptions, eligibility alignment, matching, full and selective borrowing strategies, and both asymptotic inference and randomization tests. We then demonstrate step-by-step implementation using the SelectiveIntegrative and intFRT packages. The workflow is illustrated using a synthetic lung cancer dataset included in the intFRT package that mimics the CALGB 9633 trial and ECs from the National Cancer Database. The tutorial aims to help applied statisticians conduct transparent, interpretable, and reproducible HCT analyses that improve efficiency while maintaining valid inference.

0

econ.EM 2026-07-02

Valid intervals for network densities survive group selection from data

by Eric Auerbach, Jonathan Auerbach +1 more

Post-selection inference for network structure

Two methods ensure coverage when communities or markets are identified using the observed connections themselves.

abstract click to expand

Researchers often use the density of connections between groups of agents, such as communities, blocs, or markets, to characterize the structure of a social or economic network. In many cases, these groups are selected using the network data, making conventional fixed-group inference procedures potentially invalid. To address this issue, we develop two new confidence intervals that are universally valid post-selection in the sense that they guarantee simultaneous coverage asymptotically over all pairs of groups whose relative sizes do not vanish. Our first interval builds on a strategy of \cite{berk2013valid}. Our second interval is based on a Talagrand-type concentration inequality for empirical processes. Both intervals are simple to compute and scalable to large networks, but a key technical contribution of our paper is show that only the second interval achieves the best-possible width asymptotically up to a constant factor. Three empirical illustrations show that accounting for selection can matter in practice. Some evidence for homophily in a social network and a hub-and-spoke structure in a trade network survives our correction, while evidence for disjoint market segments in a worker transition network does not.

0

stat.ME 2026-07-01

Marginal separable effects summarize causal effects for everyone

by Ruixuan Zhao, Mats Stensrud +1 more

Causal Inference for All: Marginal Estimands for Outcomes Truncated by Death

They stay interpretable and use routine longitudinal data instead of restricting to survivors or using composite summaries.

abstract click to expand

In longitudinal studies, outcomes of interest are often truncated by death, meaning that they are only observed or well-defined conditional on intercurrent events such as survival. Existing strategies face a trade-off: causally interpretable estimands, such as survivor average causal effects, target a latent subgroup, whereas while-alive and composite summaries apply to the full population but are difficult to interpret as causal effects on the non-mortality outcome. We address these challenges by introducing methodology for a new set of estimands that (i) concern the entire population, (ii) remain causally interpretable, and (iii) leverage the longitudinal data commonly available in studies with outcomes truncated by death. The set of estimands includes single-world marginal separable effects that generalize conditional separable effects to full-population summaries. We develop identification and estimation results for these estimands and apply the methodology in a reanalysis of a prostate cancer trial, highlighting how different estimands can yield different treatment conclusions.

0

stat.ME 2026-07-01

Estimator fixes normal measurement errors in quantile regression

by Mushan Li, Yanyuan Ma +1 more

Quantile regression with measurement errors

Delivers root-n consistency for linear and nonlinear models without requiring multiple quantile levels.

abstract click to expand

We devise a novel estimator for a general quantile regression model with normal measurement errors in the covariates. The method is applicable to both linear and nonlinear quantile regressions and does not impose the quantile requirement on multiple quantile levels simultaneously. We circumvent the difficulties caused by discontinuity in quantile regression through kernel smoothing, and overcome the nonlinearity inherent in quantile regression via considering extension to the complex domain and moment generating functions. We show that the resulting estimator achieves the standard root-$n$ consistency and asymptotic normality under mild conditions. The performance of the proposed method is illustrated via numerical simulations and a real data example related to Cherry Blossom times in Japan in 2024. This is the first consistent estimator in a general quantile regression problem with normal measurement errors.

0

stat.ME 2026-07-01

StaLoP boosts panel forecasts using similarity in target-local states

by Ruihang Han, Li-Hsiang Lin

Weights derived from empirical discrepancy scores replace recency assumptions, improving accuracy when patterns recur at irregular intervals

abstract click to expand

Prediction from sequential panel data is central to digital-twin modeling, where new panels arrive over time and the predictive system is updated sequentially. Existing methods often rely on temporal proximity, which can fail when similar input-output patterns recur at nonadjacent times or when recent panels differ from the target panel. We propose State-Local Prediction (StaLoP), a nonparametric dynamic panel prediction framework that utilizes information through target-local predictive compatibility. StaLoP represents panels using target-local state vectors, compares historical and target panels via empirical discrepancy scores to determine relevance weights for the target point, and combines these weights with covariate localization. Theoretical results, including bias-variance characterization, asymptotic normality, simultaneous prediction bands, and a target-local-GDF-corrected MSPE criterion for panel and model selection, are developed. Extensive simulations validate the performance of StaLoP and support its theoretical properties. Applications to sequence prediction, simulator calibration, variable selection, and county-to-county migration-flow forecasting demonstrate improved out-of-sample prediction and provide scientific insights into the underlying applications.

0

stat.ME 2026-07-01

Projection yields consistent payment estimators from macro insurance data

by Martin Bladt, Marcus Christiansen

Payment Process Estimation in Aggregated Insurance Models

Inverse-probability weighting recovers state-specific cumulative payments under truncation and censoring

abstract click to expand

Insurance payments may depend on latent micro states although only macro states and realized payments are observed. We study a sojourn-payment model for such aggregated multi-state systems under left-truncation and right-censoring. Starting from a micro-to-macro projection, we establish strong consistency and weak convergence for inverse-probability-weighted estimators of state-specific cumulative payment processes.

0

stat.ME 2026-07-01

Joint spatial model shrinks variances in ACS small-area estimates

by Zewei Kong, Paul A. Parker +1 more

Scalable Joint Modeling of Dependent Multi-Type Survey Data for Small Area Estimation

Shared random effects between binomial and Gaussian responses produce smaller posterior variances than separate univariate fits on income an

abstract click to expand

We develop a Bayesian area-level small area estimation framework that jointly models binomial and Gaussian survey responses through shared spatial random effects. This work is motivated by the American Community Survey (ACS), which provides useful information that contributes to federal funding and policy making decisions, and often yields direct estimates with large standard errors in small domains. The proposed Multi-type model borrows strength across outcomes and spatial neighbors to improve the precision of the associated estimates. For the binomial component, Polya-Gamma data augmentation yields a conditionally Gaussian representation, while spatial basis functions provide dimension reduction for high-dimensional spatial data. Together, these features lead to closed-form conditional posteriors and, thus, an efficient Gibbs sampler. Through empirical simulations, we show that the proposed joint model improves estimation precision relative to independent Univariate models. Applying the method to ACS median income and poverty rate data, we find that the proposed Multi-type model yields similar point estimates but smaller posterior variances than the corresponding Univariate models.

0

stat.AP 2026-07-01

Argo data produces OHC maps with correlated uncertainty

by Thea Sukianto, Mikael Kuusela +4 more

Locally stationary Argo ocean heat content estimates: Modeling, validation and uncertainty quantification

Locally stationary Gaussian process yields 2004-2022 anomaly fields and validated error ensembles from temperature profiles.

abstract click to expand

Argo profiling floats measure seawater temperature and salinity in the upper 2000 meters of the ocean. These floats are uniquely capable of measuring the global Ocean Heat Content (OHC), a quantity that is of central importance for understanding Earth Energy Imbalance. Yet, producing Argo-based OHC estimates with reliable uncertainties is statistically challenging due to the complex structure and large size of the Argo dataset. Here we present an end-to-end mapping and uncertainty quantification framework for Argo-based OHC estimation using state-of-the-art methods from spatio-temporal statistics. The framework is based on modeling vertically integrated Argo temperature profiles as a locally stationary Gaussian process defined over space and time. This enables us to produce computationally tractable OHC anomaly maps based on data-driven decorrelation scales estimated from the Argo observations. Our modeling choices are validated using statistical cross-validation, which demonstrates the importance of including a climatological time trend in the mean field and accounting for time in the covariance function. We quantify the uncertainty of these maps using local conditional simulation ensembles, a novel approach that leads to principled spatially and temporally correlated uncertainty quantification. A new paired cross-validation technique is presented to validate these uncertainties. The mapping framework is implemented in an open-source codebase that is designed to be modular, reproducible and extensible. To demonstrate the mapping and uncertainty quantification capabilities of this approach, we present new Argo OHC maps with uncertainties for 2004-2022 and report on various downstream climatological estimates and their uncertainties.

0

stat.ME 2026-07-01

Conformal p-values select treatment beneficiaries with FDR control

by Jiajun Liu, Ke Zhu +1 more

A Conformal Selection Framework for Individual Treatment Beneficiaries with Auxiliary External Data

RCT data anchors calibration while external data trains flexible models for patient selection

abstract click to expand

Identifying patients who are likely to benefit from a treatment is central to precision medicine and can guide follow-up trials, enrichment designs, and individualized decisions. Although randomized controlled trials (RCTs) provide evidence on efficacy, they are usually powered to estimate average treatment effects rather than patient-level benefit. Meanwhile, artificial intelligence and machine learning methods offer flexible tools for estimating heterogeneous treatment effects, especially when augmented by real-world data (RWD). However, in practice, these estimated effects are often translated into decisions through simple ranking or thresholding rules, which can ignore uncertainty and multiplicity when many patients are evaluated simultaneously. Motivated by this, we propose a model-agnostic conformal inference framework for uncertainty-aware beneficiary selection. The framework reformulates CATE-based treatment-benefit selection as a multiple-testing problem. For each candidate, we test whether the conditional treatment benefit exceeds a clinically meaningful threshold and construct a conformal p-value using RCT-based calibration. These p-values are then adjusted by the Benjamini-Hochberg procedure to control the false discovery rate (FDR) among selected beneficiaries. To improve efficiency when RCT sample sizes are limited, external data, such as RWD, can be used to train flexible treatment effect models, while conformal calibration remains anchored in the RCT data. It can be paired with conventional machine learning algorithms and emerging tabular foundation models. Simulations show that the framework maintains FDR control, with power depending on the base learner and external-data comparability. A case study in early-stage non-small-cell lung cancer illustrates how the method identifies candidate profiles with evidence of benefit from limited resection to reduce overtreatment.

0

stat.ME 2026-07-01

Screening at state level then local refinement cuts nitrogen rates

by Sakshi Arya, Abdul-Nasah Soale +1 more

Near-Optimal Nitrogen Recommendations for Precision Agriculture via Sequential Screening and Hierarchical Refinement

Method keeps yields competitive while often recommending less fertilizer than uniform state or hindsight benchmarks in corn trials.

abstract click to expand

Nitrogen fertilizer management plays a central role in balancing agricultural productivity and environmental sustainability, yet identifying optimal application strategies remains difficult because treatment responses vary substantially across locations and many fertilizer choices are statistically indistinguishable near the optimum. This paper develops a hierarchical refinement procedure, built on sequential screening, for fertilizer recommendation in multi-site experiments that explicitly accounts for spatial heterogeneity while prioritizing parsimonious, decision-oriented selection. Rather than targeting a single estimated best treatment, the proposed method first conducts sequential screening at a higher aggregation level to eliminate clearly inferior fertilizer choices and then refines recommendations locally among the surviving candidates. We study the asymptotic properties of the proposed estimators and show that it provides screening-safety guaranteed recommendations. The efficacy of the new approach is investigated through a multi-state, multi-year corn nitrogen trial. The results show that no single fertilizer regime is uniformly optimal within a state; instead, each state is associated with multiple recommended choices, and the most common recommendation typically covers only about one-third to one-half of decision units, underscoring substantial within-state heterogeneity. Representative site-level comparisons further demonstrate that the proposed method often yields lower total nitrogen recommendations than state-level or hindsight benchmarks while maintaining competitive agronomic performance.

0

stat.ME 2026-07-01

Gaussian kernel regression gains sandwich variance and bootstrap

by Marcelo R. Portela Ferreira, Eufrásio de A. Lima Neto

Statistical Inference for Gaussian Kernel Robust Regression with the gkrreg Package

Redescending M-estimator status yields closed-form standard errors plus a pairs bootstrap that re-estimates the kernel width each replicate.

abstract click to expand

The Gaussian Kernel Robust Regression method (GKRReg) is a robust regression estimator that iteratively re-weights observations via a Gaussian kernel so that outliers and leverage points receive near-zero weight, with convergence of the estimation algorithm theoretically guaranteed. Despite a thorough study of estimation, the original work leaves open the problem of statistical inference for the regression coefficients. We fill this gap with three contributions. First, we formally establish that GKRReg belongs to the family of redescending M-estimators, providing the theoretical foundation for the inferential procedures that follow. Second, we derive a closed-form analytic sandwich variance estimator based on the theory of generalised M-estimators, corresponding to the HC0 class of heteroskedasticity-robust covariance matrices; we show that a finite-sample correction analogous to HC3 requires the weighted hat matrix of the converged IRWLS step, and identify this as a direction for future work. Third, we propose a pairs bootstrap that re-estimates the kernel width hyper-parameter gamma^2 on every replicate, capturing variability that the sandwich ignores. All procedures are implemented in the R package gkrreg, which also provides four estimators for gamma^2 and an automatic data-driven selection procedure, comprehensive diagnostic plots, and six real datasets from the robust regression literature. Applications to real data sets and comparison with traditional robust regression models highlight the potential of the GKRReg and the usability of the R package.

0

stat.ME 2026-07-01

FPCA rates for manifold-indexed data depend on intrinsic dimension d

by Chang Jun Im, Jeong Min Jeon

Functional Principal Component Analysis for Manifold-Indexed Data

Geodesic kernels with volume correction yield uniform bounds whose sparse-to-dense transition is set by manifold dimension, recovering the c

abstract click to expand

Functional principal component analysis (FPCA) is a central tool for dimension reduction and covariance analysis in functional data analysis. We study FPCA for discretely observed scalar-valued functional data indexed by a compact d-dimensional Riemannian manifold M; that is, each subject is modeled as a random function from M to R. This setting is distinct from manifold-valued functional data, where the function values themselves lie on a manifold. We develop intrinsic kernel estimators for the mean and covariance functions using geodesic distances and a Riemannian volume-density correction. The proposed framework accommodates general subject-specific sampling frequencies and includes both equal-weight-per-observation and equal-weight-per-subject schemes. The uniform stochastic analysis uses VC-type empirical-process conditions for intrinsic kernel classes, together with clustered empirical-process compatibility conditions, allowing non-Lipschitz kernels under the stated assumptions. We establish uniform convergence rates for the mean and covariance estimators, Hilbert-Schmidt and operator-norm error bounds for the estimated covariance operator, and convergence rates for eigenvalues and eigenfunctions via spectral perturbation. The rates show that the sparse-to-dense transition is governed by the intrinsic dimension of the indexing manifold, reducing to the classical one-dimensional boundary when d=1. Simulations on S^1 and S^2 and a SONICOM head-related transfer function analysis illustrate the method and show modest but consistent improvements over a coordinate-based baseline when intrinsic geometry is ignored.

0

stat.ME 2026-07-01

Raw scores beat demographic corrections for some classifications

by Ignacio Gonzalez-Perez, Mats Julius Stensrud +1 more

On the choice of using raw or demographically-corrected scores

Sufficient conditions show when z-score adjustments lower accuracy on cognitive tests like the MMSE.

abstract click to expand

Demographic corrections are routinely performed in many disciplines, including psychology. Yet, there are ongoing debates about whether these corrections are appropriate and improve classification accuracy. Here, we focus on cognitive screening tests, and show that common demographic corrections, like the z-score standardization, can be detrimental for classification in some settings. Formally, we present sufficient conditions ensuring that raw scores outperform the demographically-corrected ones, and give a substantive interpretation of this result. We also investigate the claim that using demographically-corrected scores results in more fair decisions compared to using raw scores. We apply our results to the Mini-Mental State Examination in the OASIS-3 dataset.

0

stat.ME 2026-07-01

Calibration merges survey samples to sharpen regression estimates

by Yanhao Lu, Lingxiao Wang

Improving Efficiency of Regression Analyses by Integrating Data from Population-Representative Surveys: A Model-Assisted Calibration Approach

Method keeps design-based validity and works with either full records or summary statistics alone.

abstract click to expand

The increasing availability of diverse data sources has motivated great interest in data integration for improving regression efficiency. Existing data integration methods primarily focus on integrating nonprobability samples and typically assume that the integrated data sources represent the same target population. While this assumption is often difficult to justify for nonprobability samples, it is naturally satisfied when integrating probability-based surveys designed to represent a common target population. Such surveys are important research data sources because they provide representative samples and collect rich information on diverse variables, making them well suited to data integration. However, existing integration methods do not accommodate complex sampling designs. We propose model-assisted calibration methods to improve regression efficiency by integrating multiple probability-based survey samples. The proposed framework accommodates settings in which either individual-level data or only summary statistics are available from external surveys while preserving valid finite-population inference without requiring correct specification of the outcome model. We establish the design consistency of the proposed estimators and develop Taylor linearization variance estimators accounting for the complex sampling designs of both surveys. Simulation studies and an application integrating National Health and Nutrition Examination Survey and National Health Interview Survey demonstrate substantial efficiency gains while maintaining valid finite-population inference.

0

stat.ME 2026-07-01

New methods enable simultaneous bands for incomplete functional time series

by Patrick Bastian, Tim Kutta

Simultaneous Inference for Partially Observed Functional Time Series

They handle dependence and missing sensor readings to support uniform inference over the whole domain and test for trends like high pollutio

abstract click to expand

Functional data analysis (FDA) provides statistical methods for analyzing samples of time-continuous stochastic processes. Measurements often arise in the form of sensor data for a key scientific variable. The practical problem of irregular sensor disruptions has fostered interest in analyzing partially observed random functions. Specifically, this paper is motivated by a time series of intermittently missing pollution data with dependence along pollution paths and missingness patterns. To allow statistical analysis, we develop the first inference methods for dependent, partially observed functional time series. Existing methods were not appropriate for this task, because they heavily rely on the independence of the data functions. Mathematically, we model data on the space of bounded functions equipped with the supremum norm. This allows simultaneous inference across the entire functional domain, including simultaneous confidence bands -- something existing Hilbert-space-based methods cannot provide. To study non-stationary trends along the time series, we extend state-of-the-art multiscale inference methods (originally developed for scalar data) to partially observed functions. The key application of the latter methods is testing for excessive pollution levels in inner cities. Our approach combines state-of-the-art Gaussian approximations with stochastic process theory. Interestingly, it also improves existing results for fully observed functional time series by avoiding a functional CLT.

0

stat.ME 2026-07-01

Sequential designs match i.i.d

by Jiachun Li, David Simchi-Levi

Semiparametric Efficiency in Sequential Experiments: Characterization and Design via Average Propensity

Any non-anticipating experiment creates an average propensity score that bounds the precision of regular causal estimators.

abstract click to expand

Modern experiments, including evaluations of AI-enabled services and platform interventions, often depart from independent and identically distributed (i.i.d.) sampling because assignments may be adaptive, balanced across covariates, or subject to rollout constraints such as exposure, fairness, and budget limits. This paper studies the efficiency benchmark for estimating causal targets in such sequential experiments. We show that every non-anticipating design induces an average propensity score, and we establish a semiparametric lower bound: for regular locally unbiased estimators, attainable precision is bounded by the i.i.d. efficiency benchmark evaluated at this induced score. The average propensity score thereby serves as a common benchmark and design target, allowing sequential experimental design to be viewed as choosing or learning an efficient allocation rule, with operational constraints entering through the admissible set when present. We then develop implementable batched adaptive designs that approach this benchmark through two complementary mechanisms. The first uses regression adjustment based on efficient influence functions; for general smooth estimands it attains the benchmark under standard nuisance-rate conditions, while for linear functionals of outcome means it achieves a sharp second-order rate. The second uses adaptive covariate balancing to attain the same benchmark through the assignment mechanism, enabling simple moment-based estimation. Both routes require only a small number of policy updates, making them compatible with delayed feedback and easier to monitor in operational deployments. Numerical experiments and an empirical study of AI medical-assistant evaluation demonstrate the practical efficiency gains, including in multi-treatment settings. Overall, the paper provides a unified framework for characterizing and designing efficient sequential experiments.

0

stat.ME 2026-07-01

BAR penalty with smoothing yields oracle rank regression for censored data

by Suyeon Seon, Dipankar Bandyopadhyay +3 more

Censored broken adaptive ridge rank regression via induced smoothing

The estimator recovers true predictors, groups correlated ones, and supplies closed-form variances for right-censored AFT models.

abstract click to expand

Broken adaptive ridge (BAR) penalty approximates $L_0$-regularization through iterative reweighting of L2 penalties. This penalty enjoys both the oracle property and the grouping effect for highly correlated covariates, making it particularly attractive for penalized regression with complex dependence among predictors. In this paper, we develop a BAR-penalized linear rank regression method for the semiparametric accelerated failure time model with right-censored data. Computational tractability is achieved by applying induced smoothing to the nonsmooth Gehan-type rank estimating function, yielding a more stable framework for estimation and inference. For scalable penalization, we develop a cyclic coordinate descent algorithm that minimizes the penalized objective function, and estimates the regression coefficients in a coordinate-wise manner. We further extend the proposed method to more complex survival endpoints, such as multivariate partly interval-censored (PIC) data. Under mild conditions, the proposed estimator satisfies both the oracle property and the grouping effect, and the variance estimator of the informative coefficients can be derived in analytic form. Numerical studies using synthetic data compare our approach to several well-known penalties, and demonstrate its superior selection accuracy and estimation efficiency across various scenarios. Furthermore, applications to right-censored outcomes from primary biliary cirrhosis, and correlated PIC outcomes from colorectal cancer further illustrate the practical utility of the proposed method. The R package aftPenCDA for implementing the method is available on R CRAN.

0

stat.ML 2026-07-01

Hybrid swap cuts cost of dynamic GP inference on disjoint grids

by Rui-Yang Zhang, Lachlan Astfalck +3 more

Dynamic Gaussian Processes and the Vanilla-SPDE Exchange

Vanilla-SPDE Exchange uses formulation equivalence to avoid inflating spatial dimension when observations and predictions do not coincide.

abstract click to expand

Gaussian process inference is often limited by cubic computational costs, a challenge that becomes more pronounced in spatio-temporal settings where posterior inference is required over dense grids. While state-space SPDE formulations enable linear complexity in time, exact inference remains cubic in space and deteriorates further when observation locations are disjoint from the prediction locations, which inflates the number of considered spatial points. To address this, we propose the Vanilla-SPDE Exchange, which exploits an equivalence between the standard and SPDE formulations of GP inference to construct a hybrid scheme with improved computational cost. We demonstrate these gains through complexity analysis and numerical experiments.

0

stat.AP 2026-07-01

Bayesian model turns RCV poll counts into win probabilities

by Evan T. R. Rosenman, Jason Liang

Bayesian Uncertainty Quantification for Ranked Choice Voting Polls

A conjugacy relation lets the observed ballots update each candidate's chance of winning after all eliminations and transfers.

abstract click to expand

Ranked choice voting (RCV) is a popular alternative voting method in which voters are asked to list their favored candidates in preference order, rather than vote for a single candidate. When these ballots are tabulated, candidates are successively eliminated, and their votes are reallocated to each voter's next-preferred choice. The process continues until a candidate commands a majority of the active ballots and is declared the winner. As RCV gains wider adoption, the method poses novel challenges for pollsters. Unlike plurality elections, the event that a candidate wins cannot be expressed in terms of a single population parameter. Hence, the basic concept of a margin-of-error is not straightforward to define. Moreover, a candidate's ability to win may depend on both their support across the ballot and the order in which other candidates are eliminated. Existing measures of sampling uncertainty for polls of RCV elections do not clearly quantify these path-dependent outcomes. Here, we propose a simple, Bayesian framework to quantify uncertainty in polls of RCV elections. We cast the problem as one of estimating win probabilities for each leading candidate, and leverage a simple conjugacy relationship to estimate these probabilities conditional on the poll results. We include applied analyses involving two prominent ranked choice voting elections: the 2021 New York City Democratic mayoral primary, in which Eric Adams narrowly defeated Kathryn Garcia in the final round; and the 2022 special election to Alaska's U.S. House seat, in which Mary Peltola was elected despite not being a Condorcet winner. Using the cast vote records from both elections, we demonstrate some challenges of traditional frequentist uncertainty quantification in RCV polls. We also demonstrate the utility of our approach using a poll of the NYC primary obtained from the polling firm Data for Progress.

0

cs.LG 2026-07-01

Debiased ML with Bayesian priors estimates supply effects on bookings

by Yufei Wu, Daniel Schmierer +1 more

Estimating Supply Incrementality in Two-sided Marketplaces: A Causal Machine Learning Approach

The approach uses segment similarity features to produce plausible causal estimates of how extra listings change total marketplace transacti

abstract click to expand

In two-sided marketplaces with heterogeneous products, it is important to understand the causal relationship between additional supply and marketplace outcomes, such as the total quantity transacted or transaction value in the marketplace. This paper studies a causal machine learning approach to estimating this relationship across product segments. We use the Airbnb marketplace as an example, focusing on the impact of additional listing supply on total bookings, but the methodology applies to other two-sided marketplaces. Our approach combines double/debiased machine learning with a hierarchical Bayesian framework that leverages pre-existing knowledge as priors. We construct tractable and informative features for the model by leveraging measures of product segment similarity from the geospatial literature. We find that such a model provides plausible estimates of the marketplace returns to additional supply and strong out of sample performance.

0

stat.ME 2026-07-01

Hybrid PCA estimator is asymptotically normal for leading eigenvector

by Koji Tsukuda, Shun Matsuura

Hybrid principal component analysis in multivariate allometric regression

Geometric test for regression-principal-component alignment avoids instability from narrow minor-eigenvalue gaps in allometric data.

abstract click to expand

In biological data from allometry studies, the largest eigenvalue is typically dominant, and the gaps between minor eigenvalues are often narrow. Such proximity among small minor eigenvalues can lead to instability in statistics based on their corresponding eigenvectors. This study derives the asymptotic normality of the hybrid principal component analysis estimator of the leading principal eigenvector in the multivariate allometric regression model and proposes a test based on a geometric statistic for the parallelism between the regression direction and the principal component direction that avoids this instability. Using the hybrid principal component analysis framework, we analyze the well-known painted turtle carapace data and confirm previously reported results on the allometric extension relationship between female and male turtles.

0

stat.ME 2026-07-01

Clustering on spend correlations separates marketing channel effects

by Yufei Wu, Zhiying Gu +3 more

Hierarchical Clustering As a Novel Solution to the Notorious Multicollinearity Problem in Observational Causal Inference

Normalizing and demeaning data before hierarchical clustering reduces collinearity enough for Bayesian models to identify individual impacts

abstract click to expand

Multicollinearity is a long lasting challenge in observational causal inference, especially in regressions -- highly correlated independent variables make it hard to isolate their individual impacts on outcomes of interest. While common solutions such as shrinkage estimators and principal component regressions are helpful in prediction problems, a crucial limitation hinders their applicability to causal inference problems -- they cannot provide the original causal relationships. To fill the gap, we present an innovative and intuitive solution, by employing hierarchical clustering to aggregate data in a way that effectively alleviates collinearity. This method is generally applicable to causal problems featuring multicollinearity. We use a marketing application to demonstrate how and why it works. Expenditures on different advertising channels often exhibit correlations, making it exceedingly difficult to separately measure their impact. Many previous studies proposed to leverage granular cross-sectional data for better identification but, to our knowledge, none explicitly addressed multicollinearity, which undermines causal identification even with granular data. We propose to hierarchically cluster geographic units based on marketing spend correlation to reduce collinearity, and to implement a Bayesian Marketing Mix Model with cluster-level data. Such clustering happens in two steps -- we first normalize and demean geo-level data to establish a common scale and to eliminate the common trends; we then calculate pairwise distance to summarize marketing spend correlation between geos and cluster the ones with moderate to strong correlation. Both descriptive evidence and regression analysis affirm that such hierarchical clustering effectively mitigates collinearity and facilitates the separate identification of the impact of different marketing channels.

0

stat.ME 2026-06-30

Exchangeable bootstrap plus box calibration fixes hazard band coverage

by Min Lin, Grzegorz Rempala +2 more

Simultaneous confidence bands for cumulative hazard via exchangeable bootstrap and box calibration

Ratio-preserving weights and envelope calibration attain nominal levels asymptotically on the original scale with negligible overhead.

abstract click to expand

Resampling-based simultaneous confidence bands for cumulative hazard functions often undercover in finite samples with right censoring. We study two aspects of the construction that can contribute to this gap, the resampling scheme and the calibration statistic, and propose a procedure that intervenes on both. The exchangeable bootstrap reweights the numerator and the denominator of the Nelson-Aalen ratio, preserving its ratio structure. The box-calibrated discrepancy constructs lower and upper step envelopes from adjacent values of the original and resampled Nelson-Aalen estimators and measures the resulting vertical discrepancy. We establish conditional weak convergence of the exchangeable bootstrap, prove that box calibration is first-order asymptotically equivalent to grid calibration, and show that the resulting band attains nominal coverage asymptotically. The box correction uses the same bootstrap paths and event-time grid as grid calibration; after each bootstrap path is formed, it requires only an additional linear pass over the event-time grid and therefore has negligible computational overhead. In simulations across a range of hazard shapes and censoring levels, the exchangeable bootstrap with box calibration is, in most configurations, closest to nominal coverage among the methods considered. A notable consequence is a ranking reversal: the ratio-preserving exchangeable bootstrap has the lowest coverage under grid calibration, yet is usually closest to the nominal level after box calibration. A melanoma data example illustrates the practical effect on the cumulative hazard bands. The proposed procedure operates on the original cumulative-hazard scale, requires no variance-stabilizing transformation, and permits inference from time zero.

0

stat.ME 2026-06-30

Edge sampling yields finite-sample error control for network tests

by Eric Yanchenko, Jonathan P. Williams +1 more

Universal Inference for model selection on networks

Splitting edges creates an e-value that bounds type I error without asymptotics or model-specific adjustments.

abstract click to expand

Model selection and hypothesis testing are important tasks on networks. A key challenge lies in the inherent dependence in network data, as well as the fact that typically only a single realization is observed. As a result, many existing methods must be carefully tailored to specific models and only come with asymptotic theoretical guarantees. In this work, however, we propose a general model selection framework using Universal Inference, making our method widely applicable to various testing scenarios. Since Universal Inference requires two sets of data, we employ edge sampling to obtain proper networks with tractable dependence. We prove that the proposed statistic is an e-value, thus controlling the type I error rate in finite samples under nearly any hypothesis test. To our knowledge, this is the first Universal Inference-type statistic constructed from dependent splits of data as well as the first finite-sample testing guarantee for hypothesis testing on networks. We also prove that the logarithm of the test statistic diverges to positive infinity under various alternative models. On simulated and real-world networks, the proposed method performs well on tasks such as choosing the random graph model and the number of communities.

0

stat.ME 2026-06-30

Residual-on-residual regression stabilizes causal estimates

by Ashley I. Naimi, Qianhui Jin +3 more

Residual-on-Residual Regression as a Tool for Effect Estimation in Observational Data

It remains unbiased and outperforms standard methods when positivity is weak and the effect is roughly constant after confounder adjustment.

abstract click to expand

Epidemiologists increasingly use machine learning to adjust for high-dimensional confounding. Augmented inverse probability weighting (AIPW) and targeted maximum likelihood estimation (TMLE) are most widely used but may yield different results and both can become unstable under weak positivity violations. Residual-on-residual regression is a stable alternative that estimates an exposure effect encoded in a partially linear model by fitting confounder adjusted models for the outcome and exposure, then regressing outcome residuals against exposure residuals using ordinary least squares. We illustrate the approach using data from the Nulliparous Pregnancy Outcomes Study: Monitoring Mothers-to-Be (nuMoM2b; $n = 7{,}923$), estimating the association between high vegetable intake density and preeclampsia. Residual-on-residual regression, AIPW, and TMLE yielded concordant estimates, indicating a modest reduction in preeclampsia risk. In simulations, residual-on-residual regression was unbiased with near-nominal confidence interval coverage, performing comparably to AIPW and TMLE and substantially better than a misspecified parametric model when the exposure effect is approximately constant. However, in simulation settings with positivity violations, residual on residual regression outperformed AIPW and TMLE when the true effect was coded in a partially linear model. When the exposure effect is approximately constant, residual-on-residual regression is interpretable, computationally simple, and provides a triangulation strategy for observational causal inference.

0

stat.ME 2026-06-30

Dual-TV regularizers bound tensor error near minimax rate

by Wenfei Cao, Yang Chen +3 more

Exponential-Family Tensor Completion via Nonconvex Dual Total-Variation Regularization

Upper bounds reach O(n3 rt sk^2 log / n) and close the gap to the lower bound by O(sk^2 / n) for exponential-family data.

abstract click to expand

With the emergence of various tensor data, tensor completion from partial measurements has attracted widespread attention in data science and signal processing. Total Variation (TV) has been widely used as an effective regularization technique for tensor completion; however, theoretical studies on TV regularization in this context remain limited. In this work, we present a rigorous theoretical analysis of TV regularization for tensor completion. Specifically, we consider tensor completion under exponential-family noise, which generalizes the standard settings such as Gaussian and Poisson tensor completion. To handle exponential-family tensor completion, we propose a family of dual-TV (DTV) regularizers based on the transformed L1 function, which simultaneously capture sparsity and low-rank structures in the gradient tensor. Moreover, we establish the theoretical upper bounds on the recovery error of the proposed estimator. In certain cases, these upper bounds can attain the convergence order of $\mathcal{O}\big( n_3 r_t\big(\max_{k} s_k^2\big) \log\big((n_1+n_2)n_3\big) /n \big)$, and the minimax lower bound analysis is further presented to show that the upper-bounds can approach the lower bound with the gap of order $\mathcal{O}(\max_k s_k^2/max(n_1, n_2))$ up to a logarithmic factor. Finally, multiple groups of experiments on synthetic, image and video tensor data sets are conducted to support our theoretical results and demonstrate the effectiveness of our method.

0

cs.LG 2026-06-30

Personalized thresholds lift job marketplace metrics while respecting guardrails

by Yufei Wu, Zhen Yan

Personalizing Marketplace Policies with Competing Objectives and Constrained Experiments: Evidence from a Job Marketplace

Framework uses hybrid models and monotonicity extrapolation to personalize free-value policies under limited cluster experiments.

abstract click to expand

Two-sided marketplaces connect distinct user groups whose interests often conflict -- improving outcomes on one side could degrade the other side's experience. To address this challenge, we deploy an integrated framework for personalizing free-value thresholds -- a policy governing the scope of complimentary services for job listings -- across a two-sided job marketplace connecting millions of employers and job seekers. Our personalized policy delivers statistically significant and economically sizable lift in the target metric while respecting engagement guardrail constraints. Direct application of standard uplift methods proves insufficient here for two reasons. First, cross-side externalities demand multi-objective optimization: maximizing employer-side metrics risks harming job seeker engagement, with effects varying substantially across job segments. Second, marketplace interference necessitates cluster-level randomization, limiting us to few discrete treatment levels -- effectively a form of positivity violation that rules out methods designed for continuous treatments. We contribute an integrated framework with three components. Our ensemble-based hybrid ranking models target and guardrail metrics separately, cutting guardrail risk by over 10% for equivalent target gains compared to single-objective approaches. A treatment effect extrapolation method extends our estimates from limited experimental variation to untested policy levels, relying on monotonicity assumptions that we validate empirically. Finally, we present production deployment, where post-launch data confirms both extrapolation accuracy and guardrail compliance. Our deployed system demonstrates that principled methodology can enable meaningful personalization even when experiments are severely constrained and different objectives compete -- common conditions that characterize many real-world marketplaces.

0

stat.ME 2026-06-30

Cluster-level cross-fitting fixes coverage in survey TMLE

by M. Ehsan Karim

Cross-Fitted Survey-Weighted TMLE with Design-Based Variance for Causal Machine Learning

Single-fit versions cover at 0.85-0.91 while out-of-fold cluster fitting holds 0.93-0.95 once learners are flexible.

abstract click to expand

Cross-fitting is not a refinement of survey-weighted causal machine learning but, once the nuisances are flexible, what restores valid inference. We study the population average treatment effect under a stratified multistage design, estimated by a survey-aware targeted maximum likelihood estimator (TMLE) whose variance is obtained by Taylor-series linearization of the influence function, treating the primary sampling unit as the replication unit. Our central result, established in theory and simulation, is that this validity turns on cross-fitting at the cluster level. Once flexible learners cross a complexity (Donsker) boundary, single-fit survey TMLE can severely under-cover, and internal cluster-aware cross-validation does not substitute for cross-fitting; among the estimators we evaluate, only out-of-fold fitting at the cluster level restores valid coverage. In simulations spanning a many-PSU and an NHANES-like design, on a diverse ensemble the single-fit and internal cross-validation estimators cover at about 0.89-0.91 and 0.85-0.88 while the cross-fitted estimator holds at 0.93-0.95, and an aggressively grown learner drives single-fit coverage to 0.22. Two scope choices are deliberate: survey-weighted point estimation is prior work, and the nuisance product-rate condition is assumed and probed empirically. Within these conditions we prove asymptotic normality and design-consistency of the linearization variance. Four NHANES analyses and open-source software illustrate the method.

0

stat.ME 2026-06-30

Sequential shrinkage attains oracle risk for multi-source data

by Wenbo Jing, Xi Chen +3 more

Tuning-Free Efficient Estimation for Multi-Source Data via Covariance-Aware Shrinkage

Covariance information and data-driven risk reductions let the procedure combine heterogeneous sources without any tuning parameter.

abstract click to expand

Modern statistical learning problems often involve multiple related data sets, where learning efficiency on a target set can be improved by utilizing related source sets, while heterogeneity among the source sets may introduce bias. Existing approaches are limited by suboptimal performance in multi-source settings, insufficient use of covariance information, or the computational burden of tuning procedures. We propose a tuning-free and covariance-aware shrinkage framework that constructs shrinkage directions using covariance information to improve efficiency. We establish finite-sample risk bounds that yield an explicit risk-improving interval for the shrinkage size, making the procedure fully data-driven and tuning-free. When multiple source sets are available, we further propose a novel sequential algorithm that shrinks the estimator toward the sources one at a time according to their estimated risk reduction. The proposed algorithm asymptotically attains the oracle risk under mild conditions and is guaranteed to improve over the single-step shrinkage method in the literature. The framework is further extended to general smooth $M$-estimation problems via a local quadratic approximation. Numerical studies show substantial gains over competing methods, especially when the source data sets are highly heterogeneous.

0

stat.ME 2026-06-30

Net-LSW tracks time-varying dependencies on networks

by Cristian F. Jiménez-Varón, Marina I. Knight +1 more

Multiscale Dynamic Dependence Estimation over Networks

A wavelet framework encodes graph structure to recover evolving partial correlations with consistency in nonstationary data.

abstract click to expand

In numerous scientific and industrial settings, observed multivariate time series are often nonstationary in nature, i.e., comprise data whose second order properties vary over time. An additional feature of many modern datasets is that the cross-dependencies of such series are structured by an underlying network, giving rise to complex interactions between temporal dynamics and network topology. In this article we propose Locally Stationary Wavelet processes on Networks (Net-LSW), a new framework for modelling multiscale, time-varying dependencies that explicitly incorporates the network structure. Unlike traditional multivariate approaches, the Net-LSW process encodes the graph directly in the covariance structure of the process's random increments. We also introduce the concept of the local partial correlation graph, which connects edges in the graph to non-zero entries in the time- and scale-dependent dependence structure of a nonstationary process. For inference on the local cross-nodal (partial) dependence, we develop a novel subprocess-based estimation scheme and establish its desirable consistency properties. Simulation studies further demonstrate that the proposed framework accurately recovers evolving dependence structures whilst respecting the underlying graph topology. Finally, we apply our framework to daily stock price volatilities across a global bank network, demonstrating its ability to capture multiscale, highly nonstationary dependencies and identify time-varying systemic shifts during major financial shocks, including Brexit and the COVID-19 pandemic.

0

stat.ME 2026-06-30

Conditional test unifies HWE check and SNP association in GWAS

by Stefan Böhringer, Hajo Holzmann

Evaluating HWE and Association in Genome Wide Association Studies: A Unified Procedure

The procedure improves power and SNP ranking by folding equilibrium information directly into association p-values.

abstract click to expand

In genome wide association studies (GWASs) based on a case-control design, single nucleotide polymorphisms (SNPs) are typically evaluated for an association test and a Hardy-Weinberg equilibrium (HWE) goodness-of-fit test. SNPs are then excluded from analysis based on a HWE cutoff to avoid false positives. In order to avoid cutoffs based on arbitrary threshold values, we propose a conditional genotype--based test that conditions the Pearson $\chi^2$-statistic in the 3x2 contingency table on the $\chi^2$-statistic for HWE in the control group, and develop the relevant asymptotic distribution theory. We show by simulations that our test in most scenarios is more powerful than two competing retrospective procedures. Another important advantage of the proposed method is a better ranking of SNPs in GWASs as HWE is accounted for in computing p-values of SNP association. We demonstrate this effect on a data set in an alopecia study. In conclusion, our test makes separate HWE testing superfluous by providing a unified framework and strictly improves on the standard procedure in terms of power and interpretability, thereby making replication more cost effective and improving subsequent fine mapping.\par

0

math.ST 2026-06-30

Optimal confidence sets induce Choquet-risk optimal contours

by Max Raner

Efficiency of Valid Inferential Models: Choquet-risk Optimal Possibility Measures, and Direct Comparisons

For concentration penalties the risk reduces to expected contour volume, so levelwise optimality transfers directly to valid possibility mea

abstract click to expand

Valid possibilistic inferential models provide exact finite-sample calibration, but validity alone does not determine which valid procedure results in the most informative inferential summary. This paper proposes Choquet risk as a decision-theoretic criterion for comparing valid possibility measures in finite samples. Given a non-negative penalty functional, Choquet loss is defined as the Choquet integral of that penalty with respect to the data-dependent possibility measure, and Choquet risk as its sampling expectation. A key reduction expresses this risk through the nested $\alpha$-cuts of the contour, linking procedure-level efficiency to the expected performance of calibrated confidence sets. For concentration penalties, the criterion reduces to integrated expected set size, equivalently expected contour volume, so levelwise optimal confidence sets induce Choquet-risk optimal valid contours. The framework is developed along two classical routes to optimality. First, a possibilistic notion of unbiasedness is introduced and shown, under validity, to coincide with unbiasedness of the induced confidence sets and tests, allowing UMPU and most-accurate-unbiased results to be transferred to valid contours. Second, an equivariant minimax theory is developed, including a Gaussian-location result in which the Gaussian possibility contour is Choquet-risk minimax for radial distance-to-truth losses. The construction also extends confidence risk from additive confidence distributions to non-additive calibrated inferential-model output, with Choquet loss acting as a least-favourable confidence loss. Finally, the paper clarifies the penalty-dependence of efficiency comparisons and motivates invariant size criteria and divergence-based intrinsic losses connected locally to Fisher--Rao geometry.

0

math.ST 2026-06-30

Frank-Wolfe computes optimal e-values for non-convex voting tests

by Adrienne Tuynman, Timothée Mathieu

Optimal Posterior E-values with Non-Convex Parameter Sets with Applications to Voting Systems

Enables early stopping in sequential polls for Condorcet, Borda and Schulze systems while preserving validity.

abstract click to expand

We are interested in conducting political polls sequentially, so that one can stop acquiring data as soon as possible while safely yielding statistically significant results. Building off e-values, which have recently become a useful tool to create sequential testing methods, we develop a theory of posterior optimal e-values. We use voting as a convenient example on which to illustrate our method. First, we design statistical tests for Condorcet and Borda voting system, and also for Schulze voting system which we are the first to tackle statistically. Then, we study the construction of optimal sequential e-values in the deceptively simple setting of multivariate Bernoulli data, with general composite null and alternative hypothesis sets $\mathcal{H}_0$ and $\mathcal{H}_1$. We give a way to compute these e-values using an efficient Frank-Wolfe algorithm, giving a pretty general way to compute Reverse Information Projections, even when $\mathcal{H}_0$ corresponds to a non-convex parameter set. Finally, we illustrate the efficiency, both in terms of power and sample size of our method. We compare with state of the art in both simulated and real data experiments, with application to French 2022 presidential election data.

0

stat.ME 2026-06-30

Ordinal time series model estimates category spacings from data

by Anna Nalpantidi, Dimitris Karlis +1 more

Beyond Equidistant Assumptions: An Autoregressive Ordered Stereotype Model for Ordinal Time Series

The AR-OSM adds lagged responses to capture serial dependence while letting the data set distances between categories instead of assuming th

abstract click to expand

We propose an extension of the ordered stereotype model (OSM) for ordinal time series data, referred to as the Autoregressive OSM (AR-OSM). The model captures serial dependence by incorporating lagged values of the response as covariates in the systematic component. In contrast to existing regression models for ordinal time series, the AR-OSM does not assume equidistant categories, but instead allows the data to determine their relative spacing. This property makes the model particularly suitable for applications where the equidistance assumption is unrealistic. Such a case is illustrated through the analysis of infant sleep state data. Additionally, a comprehensive simulation study is conducted to assess the performance of the model under varying sample sizes and to investigate how parameter values influence the induced serial dependence structure.

0

cs.AI 2026-06-30

Personal decision theory optimal under population enforcement metric

by Arvid Sjölander

A causal modeling perspective on decision theory

Causal models show it maximizes utility when a whole population is made to follow it.

abstract click to expand

Decision theory provides a formal framework for how agents should make choices under uncertainty, drawing on ideas from philosophy, probability, and causality. Despite significant progress, the field still lacks a unified modeling language, and key concepts - such as the distinction between subjective and objective elements, or what it means for a decision theory to perform well - are often left implicit. This can make it difficult to evaluate and compare competing theories, particularly in controversial cases. In this paper, we address these issues by introducing a formal framework for decision theory based on nonparametric structural equation models (NPSEMs), a well-established tool in causal inference. NPSEMs provide a unified foundation for representing agents, counterfactuals, and causal relationships, allowing for unambiguous definitions of EDT and CDT. Building on this foundation, we propose a novel decision theory - personal decision theory - which instructs agents to maximize a subjective model of their own counterfactual utility. We introduce a formal performance metric based on hypothetical interventions that enforce a given decision theory across a population - such as might be achieved through education or policy -- and show that, under certain assumptions, personal decision theory is optimal with respect to this metric. Throughout, we use the smoking lesion problem as a running example and conclude with a formal analysis of Newcomb's problem. Our aim is to provide decision theory with a clearer modeling language and firmer evaluative ground, thereby enabling more rigorous comparisons and facilitating conceptual progress in the field.

0

physics.data-an 2026-06-30

Squeals flag poor model fits while users drag curves

by Andrew Gelman, Andrew H. Jaffe +2 more

The Squealer: Sensification of model exploration and model misfit

An unpleasant sound grows louder as the curve moves away from the data, turning visual inspection into an immediate sensory check.

abstract click to expand

We introduce a method for visual and auditory feedback when exploring the fit of a model to data. Starting with a best-fit curve fit to data, the user can drag the curve to a new position and the computer will emit a squeal, becoming louder and more unpleasant as the discrepancy between curve and data increases. We demonstrate with four examples: a two-parameter curve fit to golf putting data, a four-parameter curve fit to dilution assays, a fit to cosmological data sensitive to the parameters of the Big Bang model, and a nonparametric Gaussian process fit to temperature readings.

0

stat.ME 2026-06-30

New downscaling method matches kriging accuracy at far lower compute cost

by Daisuke Murakami, Yongwan Chun +2 more

Scalable coarse-to-fine spatial downscaling

Coarse-to-fine synthesis of local models enables fast spatial predictions for large datasets without covariance inversion.

abstract click to expand

This study proposes coarse-to-fine downscaling (CF-DS), a scalable spatial downscaling method extending coarse-to-fine spatial modeling. Unlike conventional spatial-statistical downscaling methods such as area-to-point kriging, CF-DS does not require covariance matrix inversion or likelihood evaluation. Instead, it represents latent spatial processes through the synthesis of multi-scale local models, substantially reducing computational cost while approximately satisfying the aggregation constraint. Monte Carlo experiments show that CF-DS achieves predictive accuracy comparable to area-to-point kriging with dramatically shorter computation times, particularly for large datasets. An application to downscaling electricity consumption in the Tokyo metropolitan area further demonstrates its practical usefulness. The results suggest that CF-DS provides an efficient alternative for large-scale spatial downscaling problems. CF-DS is implemented in an R package spCF (https://cran.r-project.org/web/packages/spCF/index.html).

0

stat.ME 2026-06-30

Historical data cuts bias and variance in model evaluations

by Xinrui Ruan, Zhenyu Zhao +5 more

HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data

HERO calibrates noisy silver labels from past rounds and anchors to precise covariates for more reliable current tests.

abstract click to expand

Reliable generative AI models critically rely on expert human annotations to evaluate output quality, yet these "gold" labels are expensive to collect and limited in quantity. Organizations thus often turn to collecting vast but noisy "silver" labels from crowdsourced workers or vendor annotators as proxies for gold labels. Because gold remains the evaluation target, naively aggregating noisy silver labels may introduce bias, and estimators built on sparsely observed gold labels may have high variance to resolve the model performance gaps that guide practical decisions. Model evaluation has become an ongoing operational practice rather than a one-time exercise, with evaluation rounds repeating across model versions, releases, and content domains. A natural question is whether the previous historical evaluation data can be used to improve each new round of evaluation. We introduce HERO (History Enhanced RObust model evaluation), a novel framework that uses historical data to suppress bias (improve reliability) and reduce variance (improve sensitivity) in model performance evaluation. HERO calibrates silver labelers' performance learned from historical gold annotations, and stabilizes the resulting estimator by anchoring it to covariate information measured with high precision in the historical data. HERO can be broadly applied across multiple common evaluation tasks, and remains valid when only a subset of historical labelers appears in the current round. We establish conditions under which the bias and variance reductions hold, showcase HERO's performance in simulation studies, and demonstrate its effectiveness on real-world model evaluation benchmarking datasets.

0

stat.ME 2026-06-30

Symmetric noise enables valid tests after data-driven selection

by Ameer Dharamshi, Runjia Zou +1 more

Testing hypotheses via orthogonalization

Partition data by adding and subtracting symmetric noise, then test if orthogonalization succeeds to validate the null without pre-specifyin

abstract click to expand

Classical hypothesis testing frameworks break down in contemporary settings in which null hypotheses are increasingly abstract, the same data are used to both generate and test hypotheses, and minimal assumptions about the underlying data are made. In this work, we propose a new framework for conducting valid hypothesis tests in broad contexts. We propose to add and subtract external noise generated from a symmetric shift-family to our data, $X$, to partition it into two pieces, $X^{(1)}$ and $X^{(2)}$. We provide a generic strategy for orthogonalizing $X^{(2)}$ against $X^{(1)}$ under the null hypothesis $H_0$, then show that testing whether the orthogonalization was successful provides a valid test of $H_0$ under mild assumptions. Remarkably, this framework extends naturally to the post-selection inference setting: we simply select a hypothesis on $X^{(1)}$, then perform orthogonalization under the selected null. As our approach neither requires pre-specification of the selection mechanism, nor is restricted to a small class of data-generating distributions, it dramatically expands the settings for which valid post-selection inference can be conducted. We showcase the flexibility of our proposal in several case studies involving challenging pre-specified null hypotheses and post-selection inference scenarios.

0

stat.ME 2026-06-29

Summary statistics enable privacy-preserving transfer for single-index models

by Ye Tian

Multi-Source Transfer Learning of Sparse Single-Index Models

Generalized Stein's lemma summaries allow index estimation across domains and nonlinear fitting on target data alone.

abstract click to expand

Transfer learning leverages knowledge from related source domains to improve learning in a target domain. Recent theoretical advances cover a broad range of regression settings within (generalized) linear models. Despite their diversity, these methods share two common constraints: they assume a known link function or linear structure and require direct access to raw source data. To move beyond these constraints, we propose a source-data-free transfer learning framework based on the single-index model (SIM). Instead of requiring raw source data, our method transfers only summary statistics derived from a generalized Stein's lemma in a one-time communication. This design preserves privacy and avoids side effects caused by dissimilarities of unknown nonlinear link functions across domains. To capture flexible, unknown nonlinearity, we employ a multilayer perceptron guided by the pre-estimated index from the transferred statistics, which significantly mitigates overfitting. Extensive experiments on synthetic data and a real-world application demonstrate consistent improvements over existing (generalized) linear model-based approaches. The proposed framework thus offers a practical, privacy-preserving, and nonlinear-adaptive solution for transfer learning.

0

stat.ME 2026-06-29

Shared blocks enable consistent class graph recovery

by Seunghyun Lee, Yuqi Gu

Beyond Local Independence: High-Dimensional Latent Class Graphical Models with Shared Block Structure

Three-step estimator with error bounds recovers latent classes, shared partitions, and class-specific dependencies from high-dimensional ord

abstract click to expand

Latent class models are central tools for multivariate categorical data from heterogeneous populations, but their standard local-independence assumption is often unrealistic in modern high-dimensional applications. We propose a high-dimensional latent class graphical model for ordinal responses with block-structured local dependence. The model retains the interpretability and parsimony of classical latent class analysis by imposing a shared block partition of variables, while allowing class-specific graphical dependence within each block. We develop a scalable three-step estimator that first recovers latent classes by spectral clustering of a flattened response matrix, then estimates class-specific latent covariance matrices and aggregates them to recover the shared block partition, and finally estimates sparse within-block precision matrices. We establish finite-sample error bounds for clustering, covariance estimation, block recovery, and precision-matrix estimation, yielding end-to-end consistency of all model components under high-dimensional scaling. Simulations demonstrate accurate recovery of latent classes, the shared block partition, and class-specific dependence graphs with scalable computation. Applications to American National Election Studies survey data and HapMap3 genotype data show that the method uncovers interpretable local dependence structures while accounting for latent heterogeneity.

0

stat.ME 2026-06-29

Mixture models distinguish mild from gross anomalies in circular data

by Antonio Punzo, Andriëtte Bekker +3 more

Modelling and detecting mild and gross anomalies in circular data via double-contaminated models

A double-contaminated model combines reference, less concentrated, and uniform components to classify observations and measure anomaly preva

abstract click to expand

In this paper, we propose a model-based framework to robustify inference for circular data in the presence of anomalous observations, distinguishing between mild and gross anomalies. Starting from a unimodal and symmetric reference model on $[0,2\pi)$, parametrized by a mean direction and concentration, we construct a family of finite mixtures: a gross-anomaly model obtained by adding a circular uniform component; a mild-anomaly (contaminated) model obtained by mixing the reference distribution with a less concentrated version sharing the same mean direction; and a general three-component specification combining both models, the double-contaminated model. Posterior component probabilities provide an automatic classification of observations without ad hoc thresholds, while the mixing weights yield interpretable measures of anomaly prevalence and dispersion inflation. For illustration, we consider two classical circular reference distributions, the wrapped normal and von Mises. The methodology is evaluated through an extensive simulation study and three real-data applications involving animal movement directions and wind directions. The results indicate that jointly modelling mild and gross departures improves model fit and yields an informative decomposition of the directional data, demonstrating that mixture-based robustness is valuable not only for anomaly detection but also for the interpretation and the identification of latent structure in directional data.

0

stat.ME 2026-06-29

Bayesian model segments satellite images without local labels

by Bao Khanh Nguyen, Iain Cameron +2 more

Scalable Bayesian Spatial Mixture Modelling for Remote Sensing Image Segmentation

POTTERS extends the Potts model with external priors and variational inference for scalable segmentation and uncertainty in new regions.

abstract click to expand

Accurate and scalable land cover classification is essential for global conservation monitoring and policy-making. While remote sensing images provide a cost-effective alternative to ground surveys, current methods often lack principled uncertainty quantification and require substantial labelled data, limiting their usability and reliability in new regions with distribution shifts. We propose a Bayesian spatial mixture modelling approach for image segmentation, extending the classical Potts model by allowing for a generalised spatial dependence structure and incorporating informative priors estimated from pre-existing labelled data. Our framework, called POTTERS (Potts Model for Enhanced Remote Sensing), enables robust uncertainty quantification, accounts for class interactions, and can detect new clusters in the target region of interest. Crucially, our model does not require labelled data from the target region; instead, it incorporates prior information about the labels from pre-existing externally labelled images. To ensure scalability to large remote sensing images, we develop an efficient variational inference algorithm for posterior approximation. We demonstrate the benefits of our approach in simulation studies and apply it to land cover classification in a case study in Scotland, leveraging publicly available remote sensing data from England.

0

cs.LG 2026-06-29

Model tracks Sri Lankan vegetable price surges at 86 percent accuracy on unseen 2024 data

by Ranuga Weerasekara, Heshan Nethmina +7 more

When Prices Double in a Week: Forecasting of Agricultural Volatility in Import-Isolated Markets

Unified ensemble uses weather, diesel, and exchange rates plus two cultivation seasons to maintain performance without retraining.

abstract click to expand

Vegetable prices in Sri Lanka are highly volatile because the market is largely import-isolated, so supply disruptions quickly drive prices up. This study develops a machine learning framework to forecast such volatility by incorporating supply-chain-aware features and explicitly modelling the country's two cultivation seasons, Maha (October-April) and Yala (May-September). An integrated dataset was constructed by combining retail and farmer-gate prices with origin-aligned weather variables, diesel costs, and exchange rates across 12 vegetable varieties and 14 market centres from 2013 to 2019. A gradient-boosted ensemble model (XGBoost and LightGBM) was trained and optimised using Optuna, and unified and season-specific configurations were compared. Results show that season-specific models improve within-season fit, with the Yala-specific model achieving the highest R2 of 0.9420 (95% CI [0.690, 1.000]), while the unified model delivers the best overall predictive accuracy of 90.84% (95% CI [88.34%, 91.52%]) and an R2 of 0.9281 (95% CI [0.760, 1.000]). Notably, the unified model maintains 85.96% accuracy on a completely unseen 2024 hyperinflationary period without retraining, successfully tracking major price surges. These findings suggest that agricultural price movements in import-constrained markets are meaningfully predictable when models capture supply-chain dynamics, offering practical value for early warning and decision making by farmers, traders, and policymakers. Existing studies on Sri Lankan vegetable prices are confined to Autoregressive Integrated Moving Average (ARIMA) and Generalized Autoregressive Conditional Heteroskedasticity (GARCH) applied to single markets, with no supply-chain features, seasonal segmentation, or cross-regime validation.

0

stat.ME 2026-06-29

Multivariate BART obtains first contraction rates with joint residual dependence

by Soham Ghosh, Sameer K. Deshpande

Multivariate Varying-Coefficient BART with Graphical Horseshoe Priors

Independent tree ensembles per coefficient plus a graphical horseshoe prior allow near-minimax adaptation while recovering sparse outcome ne

abstract click to expand

Modern multivariate regression problems involve several related outcomes whose regression effects are not only nonlinear, heterogeneous, and outcome-specific, but also where the residual dependence among outcomes is scientifically meaningful. Existing multivariate Bayesian tree-based methods typically address only part of this problem: some impose substantial sharing of tree architecture across outcomes, which is overly restrictive when responses depend on distinct predictors or effect modifiers, while others accommodate residual dependence but retain simpler mean structures. This paper develops multiVCBART, a multivariate varying-coefficient Bayesian additive regression tree framework that jointly models flexible outcome-specific coefficient surfaces and a sparse residual precision matrix. Each entry of the coefficient matrix $B(x)$ is represented by an independent BART ensemble, allowing predictor effects to vary nonlinearly with modifiers $x$ across outcomes, while a Graphical Horseshoe prior on the precision matrix $\Omega$ captures parsimonious residual conditional dependence. To permit efficient computation, we introduce a sampler that reduces the multivariate Gaussian likelihood to a sequence of scalar pseudo-response updates, decoupling the tree backfitting from the Graphical Horseshoe step. Theoretically, we establish the first posterior contraction rates for a multivariate BART model with jointly estimated residual dependence, proving near-minimax adaptation to underlying smoothness and structural sparsity. Empirically, multiVCBART outperforms existing multivariate tree models and Bayesian SUR competitors on sparse, high-dimensional datasets. Finally, in a re-analysis of the Genomics of Drug Sensitivity in Cancer dataset, our method identifies distinct biomarker signals and recovers a coherent residual pharmacologic network.

0

math.ST 2026-06-29

Mixture posteriors adapt to true K with extra mass vanishing at n^{-1/2}

by Filippo Ascolani

Posterior concentration and adaptation of the mixing measure in Dirichlet process mixtures

When data follow a finite mixture, the Dirichlet process concentrates on the correct number of components, yielding nearly optimal contracti

abstract click to expand

We study the asymptotic properties of the posterior on the latent space for infinite mixtures driven by a Dirichlet process, both in terms of mixing measure and clustering behaviour. In the well-specified regime, where the data are generated by a finite mixture of location densities, we show that the posterior is adaptive to the true number of components $K$: indeed the cumulative mass assigned to weights of the stick-breaking representation beyond the $K$-th one vanishes as $n^{-1/2}$, up to terms growing slower than any polynomial. This also implies a nearly optimal posterior contraction rate for the mixing measure in Wasserstein distance. A remarkable phase transition underlies this result: approximating the mixing measure to any precision finer than $n^{-1/2}$ requires a number of components growing logarithmically with the sample size. We show that this has a profound impact on the clustering behaviour: the number of clusters grows logarithmically, as in the prior case, but the proportion of observations outside the $K$ largest clusters vanishes polynomially fast. Finally, we turn these results into posterior guarantees for truncation-based approximations: while any truncation with at least $K$ elements recovers the optimal contraction rates for both density and mixing measure, $\mathcal{O}(\log n)$ components are both necessary and sufficient to reproduce the clustering of the exact posterior.

0

stat.ME 2026-06-29

Panel flow matching pools sparse longitudinal data into full distributions

by Jianbin Tan, Pixu Shi +1 more

Panel Flow Matching: A Generative Approach to Learning Distributions of Longitudinal Data

Forward flow-matching plus backward kernel fitting yields density estimates and supports completion and classification without dimension red

abstract click to expand

Learning distributions of longitudinal data is central to tasks such as visualization, completion, classification, and synthetic data generation, but it remains statistically challenging because longitudinal observations are often irregular, sparse, and collected from only a limited number of subjects. To address this, we develop a novel generative framework, termed panel flow matching (PFM), for learning longitudinal distributions by pooling information across time via a continuous panel flow model. PFM combines a forward flow-matching step with a backward kernel-fitting step, yielding a flexible and data-adaptive approach for capturing complex distributional structures. We apply PFM to estimate panel densities, namely the cross-sectional densities of longitudinal data, and establish statistical guarantees under irregular and sparse sampling designs. Under this, PFM naturally supports tasks including longitudinal completion, synthetic data generation, and classification, without requiring a preliminary dimension-reduction step to handle data irregularity. Extensive simulations demonstrate that PFM outperforms existing methods across these tasks. We further apply PFM to a vaginal microbiome longitudinal dataset from 188 pregnancies labeled as term or preterm, where it improves classification accuracy and reveals time-varying distributional differences between the two groups.

0

stat.CO 2026-06-29

spca R package produces uncorrelated sparse principal components

by Giovanni Maria Merola

spca: An R package to Compute Least Squares Sparse Principal Components

LS-SPCA maximizes variance explained while keeping the components uncorrelated and close to ordinary principal components.

abstract click to expand

This paper introduces the R package spca, which provides a computational framework for least squares sparse principal component analysis (LS-SPCA). Unlike other SPCA methods, LS-SPCA generates uncorrelated sparse principal components (sPCs) that effectively maximize the explained variance while maintaining strong correlations with standard principal components (PCs). The framework also includes more computationally efficient variants that produce mildly correlated sPCs, which often have lower cardinality while explaining equal or greater variance than the LS-SPCA optimal sPCs. The spca package is built on an efficient C++ backend for matrix computations, with distinct engines for tall and fat matrices, and a flexible R frontend. The user interface offers several options for computing sPCs, such as deciding whether sparsification should stop when a threshold for cumulative variance explained or R2 with the PCs is reached, and choosing between simple forward selection, stepwise forward selection, or backward elimination for variable selection. In addition to the print(), summary(), and plot() methods, the package includes tools for comparing different "spca" solutions, grouping sparse loadings, and representing foreign SPCA solutions as "spca" objects. This article demonstrates with real datasets the use of the package in a typical LS-SPCA workflow and briefly contrasts LS-SPCA with conventional SPCA solutions . Then it compares different LS-SPCA solutions obtained from the dataset. Finally, the performance of spca on large tall and fat matrices is discussed, showing that spca offers a computationally efficient alternative for computing interpretable sPCs.

0