Duality in Financial Theory
The Illusion of Closed-System Optimization and the Normative Imperative of Markets
1 Introduction
Modern neoclassical finance is built upon an epistemological sleight of hand: the forced reduction of an inherently open, evolutionary socioeconomic system into a mathematically tractable closed, conservative mechanical system. By borrowing the formalism of nineteenth-century thermodynamics and analytical mechanics, orthodox financial theory models capital markets as self-contained, ergodic spaces governed by convex optimization, complete risk-sharing, and spontaneous no-arbitrage equilibria. In this idealized architecture, asset prices emerge as immutable physical constants, and market efficiency is treated as a law of nature.
This ontological reductionism fundamentally mischaracterizes social reality. Unlike physical systems bounded by immutable physical laws, financial markets are open systems characterized by path-dependent institutions, radical uncertainty, endogenous power asymmetries, and non-ergodic structural shifts. When mainstream models encounter empirical breakdowns—such as the persistent outperformance of market-dominating mega-caps, the failure of portfolio diversification, or long-standing anomalies like the Equity Premium and Risk-Free Rate puzzles—they diagnose them as mere “puzzles” or add ad-hoc statistical factors, rather than acknowledging the bankruptcy of their closed-system premises.
This paper provides a unified critical synthesis across five foundational pillars:
- The Mathematical Apex of Duality: We formalize the exact mathematical duality between microeconomic convex optimization (the primal problem) and macroeconomic no-arbitrage pricing kernels (the dual problem), clarifying the precise boundaries of existence, uniqueness, and closed-form tractability.
- The Fragility of the Risk-Return Trade-Off: We demonstrate how industrial and financial market concentration destroys the fundamental mechanism of portfolio diversification within the Capital Asset Pricing Model (CAPM).
- The Pathology of Linear Factor Pricing: We expose how multi-factor frameworks (APT, Fama-French) confuse monopoly rent extraction and institutional insulation with compensation for systematic risk-bearing.
- The Mirage of Canonical Puzzles: We prove that the Equity Premium and Risk-Free Rate puzzles are not genuine macroeconomic anomalies, but structural artifacts generated by imposing homogeneous representative-agent Euler equations upon profoundly unequal, participation-constrained populations.
- The Normative Imperative of Markets: We establish that free-market conditions—Perfect Competition, No-Arbitrage, and Market Efficiency—are not spontaneous natural equilibria. Unlike natural sciences, social sciences deal with reflexive, self-defined institutions. Legitimate market conditions must be deliberately constructed, defended, and continually perfected through collective human action and institutional design.
2 Duality Between Asset Prices and Optimal Quantities
The mathematical core of modern asset pricing rests upon a fundamental duality connecting the microeconomic primal problem of expected utility maximization and the macroeconomic dual problem of no-arbitrage pricing.
Market completeness ensures the uniqueness of the Stochastic Discount Factor (SDF) and the Equivalent Martingale Measure (EMM). Concurrently, from the investor’s perspective, the uniqueness of the optimal asset allocation \(q^*\) is governed by strict concavity of the utility function (\(u'' < 0\)) and positive-definiteness of the covariance matrix (\(\Sigma \succ 0\)). The representative agent hypothesis serves as the structural bridge connecting these two domains, reconciling market-clearing equilibrium prices through a single aggregate objective.
This duality decomposes into three structural tiers: existence, uniqueness, and closed-form analytical solvability.
2.1 Structural Decomposition
| Structural Layer | Mathematical Object | Market / Dual Perspective | Investor / Primal Perspective |
|---|---|---|---|
| 1. Existence | Admissible domains of \((p, q)\) | No-Arbitrage \(\iff\) Existence of at least one positive SDF \(M > 0\) (FTAP 1) | Finite maximizer \(q^*\) under compact budget set or Inada conditions |
| 2. Uniqueness | Singleton solution properties | Market Completeness (\(\text{Rank}(X) = S\)) \(\iff\) Unique EMM / SDF (FTAP 2) | Price-taker + Strict risk aversion (\(u'' < 0\)) + Non-degeneracy (\(\Sigma \succ 0\)) |
| 3. Closed-Form | Algebraic solvability | Linear factor pricing models (Affine SDF specifications) | Matched utility-distribution pairs (Quadratic, CARA-Normal, continuous HJB) |
2.2 Existence: No-Arbitrage and Feasible Utility Maximization
By the First Fundamental Theorem of Asset Pricing (FTAP 1), the absence of arbitrage opportunities in frictionless financial markets is mathematically equivalent to the existence of at least one strictly positive stochastic discount factor \(M > 0\), or equivalently, a risk-neutral martingale measure \(\mathbb{Q}\) equivalent to the physical measure \(\mathbb{P}\) (\(\mathbb{Q} \sim \mathbb{P}\)).
Let \(X\) denote the \(S \times N\) payoff matrix across \(S\) states for \(N\) assets. The no-arbitrage condition guarantees that the price vector \(p\) passes through the interior of the asset payoff cone:
\[ p = \mathbb{E}[M X] = \frac{1}{1 + r_f} \mathbb{E}^\mathbb{Q}[X] \]
For the individual investor, the no-arbitrage condition prevents the attainment of infinite expected utility via unbounded leverage (\(\sup \mathbb{E}[u(W)] = \infty\)). Guaranteeing the existence of a finite maximizer \(q^*\) requires that the objective function be bounded from above and that the budget set be compact, or that the utility function impose asymptotic Inada conditions (\(\lim_{W \to 0} u'(W) = \infty\)) over positive terminal wealth.
2.3 Uniqueness: Market Completeness vs. Portfolio Uniqueness
Market completeness and individual portfolio uniqueness operate in distinct mathematical spaces and must not be conflated:
- Market Pricing Perspective (Dual): According to the Second Fundamental Theorem of Asset Pricing (FTAP 2), an arbitrage-free market possesses a unique equivalent martingale measure \(\mathbb{Q}\) if and only if the market is complete. Market completeness requires that asset payoffs span the entire state space (\(\text{Rank}(X) = S\)). When markets are incomplete (\(N < S\)), there exist infinitely many admissible SDFs, and contingent claims cannot be uniquely priced by no-arbitrage restrictions alone.
- Investor Perspective (Primal): The uniqueness of an individual investor’s optimal portfolio \(q^*\) does not require market completeness (\(\text{Rank}(X) = S\)). Instead, it requires non-degeneracy of the tradable asset menu (\(\text{Rank}(X) = N\)):
- Price-Taking Behavior: The investor treats asset prices \(p\) as exogenous parameters.
- Strict Risk Aversion: The utility function satisfies \(u''(W) < 0\) globally.
- Absence of Multicollinearity: The return covariance matrix \(\Sigma\) must be positive-definite (\(\Sigma \succ 0\)):
\[ \Sigma \succ 0 \iff \nexists \, v \neq 0 \quad \text{s.t.} \quad v^T R = c \quad (\text{a.s.}) \]
If an asset is redundant (collinear with others), multiple optimal portfolios \(q^*\) exist even in a fully complete market. Conversely, even in severely incomplete markets (\(N \ll S\)), if tradable assets exhibit no linear redundancies (\(\Sigma \succ 0\)) and the investor possesses strictly concave utility, the optimal allocation \(q^*\) is uniquely determined.
2.4 The Role of the Representative Agent
The representative agent hypothesis is not required for portfolio uniqueness at the individual level. Rather, it is invoked in general equilibrium asset pricing to close the system, clearing markets endogenously and binding market prices \(p\) directly to an aggregate SDF:
\[ M = \beta \frac{u_{rep}'(W_M)}{u_{rep}'(W_{M,0})} \]
where \(\beta\) is the subjective time-discount factor, linked to the risk-free rate via \(1 = (1+r_f)\mathbb{E}[M]\).
2.5 Closed-Form Analytical Tractability
While uniqueness of \(q^*\) is ensured by the second-order condition (negative definiteness of the Hessian \(\nabla^2_q \mathbb{E}[u(W_1)] \prec 0\)), expressing this solution in closed form requires linearizing the first-order condition:
\[ \mathbb{E}\left[u'\!\left(W_0(1+r_f) + q^T (X - (1+r_f)p)\right) \cdot (X - (1+r_f)p)\right] = 0 \]
This requires highly restrictive structural pairings:
- Quadratic Utility: \(u'(W)\) is linear in \(W\), collapsing the expectation into standard mean-variance matrix algebra.
- CARA-Normal: Exponential utility combined with Gaussian payoffs evaluates through the moment-generating function.
- Continuous-Time CRRA: Utilizing the infinitesimal generator of Brownian motion transforms the integral into a second-order Hamilton-Jacobi-Bellman (HJB) PDE, yielding constant allocation fractions (Merton’s portfolio problem).
3 Challenging the Risk-Return Framework
The Capital Asset Pricing Model (CAPM) (Sharpe 1964; Lintner 1965; Mossin 1966) collapses the general duality framework into an operational single-factor equilibrium. It partitions risk into systematic risk, which cannot be eliminated through diversification, and unsystematic risk, which is asset-specific and diversified away:
\[ \mathbb{E}[R_i] = R_f + \beta_i (\mathbb{E}[R_m] - R_f), \quad \beta_i = \frac{\operatorname{Cov}(R_i, R_m)}{\operatorname{Var}(R_m)} \]
3.1 The Mathematical Foundation of Diversification
Consider an equally weighted portfolio of \(n\) assets. Portfolio variance \(\sigma_p^2\) is given by:
\[ \sigma_p^2 = \frac{1}{n} \bar{\sigma}^2 + \frac{n-1}{n} \bar{\rho} \bar{\sigma}^2 \]
where \(\bar{\sigma}^2\) is the average asset variance and \(\bar{\rho}\) is the average pairwise correlation. As \(n \to \infty\), portfolio risk converges entirely to the average covariance \(\bar{\rho} \bar{\sigma}^2\) (Elton and Gruber 1977).
3.2 The Breakdown Under Market Concentration
The egalitarian risk-sharing narrative of CAPM presupposes a dispersed, competitive market structure. However, contemporary equity markets exhibit unprecedented industrial and financial concentration (Grullon et al. 2019; Neuhann and Sockin 2024).
When a handful of mega-capitalization firms dominate index weights, the core mechanism of diversification breaks down:
- Correlation Elevation: As dominant platform firms capture systemic market share, pairwise correlation \(\bar{\rho}\) rises, diminishing the variance-reduction benefits of increasing portfolio size \(n\).
- Compression of the Payoff Space: In standard arbitrage-free pricing, assets on the upper mean-variance frontier correlate perfectly with the SDF (Cochrane 2009). In concentrated markets, mega-cap firms with durable economic moats (Bustamante and Donangelo 2017) act as the principal components of the entire economy, compressing the state space.
- From Risk-Bearing to Rent Extraction: Dominant firms do not earn returns by exposing capital to operational volatility; they earn economic rents through market power, regulatory capture, and network lock-in (Hou and Robinson 2006; Edmans 2009). Secondary market trading becomes a zero-sum exercise in capturing concentrated rents, directly contradicting CAPM’s foundational premise that higher returns reflect compensation for bearing productive economic risk.
4 The Failure of Linear Factor Models and Rent Extraction
The conventional response to asset pricing anomalies has been the progressive addition of ad-hoc linear factors—from the Fama-French size and value factors (Fama and French 1993) to multi-factor models incorporating profitability, investment, and momentum. This approach represents an epistemological failure: it attempts to preserve the linear, closed-system paradigm by mechanically appending explanatory variables:
\[ \mathbb{E}[R_i] = R_f + \sum_{k=1}^K \beta_{i,k} F_k + \alpha_i \]
4.1 Statistical Pathologies
Multi-factor expansions suffer from fatal methodological weaknesses: 1. Multicollinearity and Instability: Variables proxying market power, intangible assets, network externalities, and institutional insulation exhibit extreme mutual correlation. This generates severe variance inflation, erratic coefficient estimates, and invalid inferences. 2. The Bias-Variance Dilemma and P-Hacking: Adding factors mechanically improves in-sample explanatory power (\(R^2\)) while disastrously degrading out-of-sample predictive validity:
\[ \text{MSE}(\hat{y}) = \text{Bias}^2(\hat{y}) + \operatorname{Var}(\hat{y}) + \sigma^2 \]
Given the non-ergodic nature of financial time series, factor models overfit historical noise rather than identifying invariant economic laws. 3. Parameter Non-Stationarity: Linear specifications assume invariant factor loadings (\(\beta_{i,k,t} = \beta_{i,k}\)). In reality, institutional shifts, antitrust interventions, and technological disruptions generate structural regime breaks, rendering historical betas obsolete.
4.2 The Inverse Risk-Return Paradox of Mega-Caps
Decomposing beta into correlation and relative volatility exposes a profound empirical contradiction:
\[ \beta_i = \rho_{i,m} \frac{\sigma_i}{\sigma_m} \]
Comparing dominant mega-caps to non-mega-cap equities reveals that both groups maintain high market correlations (\(\rho_{i,m} \approx 1\)). However, mega-caps exhibit systematically lower idiosyncratic volatility (\(\sigma_i^{\text{mega}} < \sigma_i^{\text{non-mega}}\)) due to diversified revenue streams, pricing power, and implicit “too-big-to-fail” institutional backstops.
Consequently, dominant firms generate higher returns with lower fundamental risk, inverting the textbook security market line. Returns in contemporary capitalism flow not as a reward for bearing vulnerability, but as an extraction of structural privilege.
5 Puzzles as Artifacts of Closed-System Homogeneity
For decades, financial economics has struggled with two foundational anomalies: the Equity Premium Puzzle (EPP) (Mehra and Prescott 1985) and the Risk-Free Rate Puzzle (RFRP) (Weil 1989). Both originate from the consumption-based capital asset pricing Euler equation:
\[ \mathbb{E}_t \left[ \beta \left( \frac{C_{t+1}}{C_t} \right)^{-\gamma} R_{i,t+1} \right] = 1 \]
Approximating under log-normality yields:
\[ \mathbb{E}[R_e - R_f] \approx \gamma \sigma_c^2 \]
\[ r_f \approx \delta + \gamma g_c - \frac{1}{2} \gamma (\gamma + 1) \sigma_c^2 \]
Because aggregate consumption growth variance \(\sigma_c^2\) is small (1%–2% annually), reconciling the historical 6%–8% equity premium demands implausibly extreme risk aversion (\(\gamma > 20\)). Simultaneously, plausible risk aversion parameters predict real risk-free rates of 5%–8%, far exceeding the empirical 1%–2% average.
These anomalies are not genuine economic puzzles. They are mathematical artifacts created by forcing a homogeneous representative agent onto an economy characterized by extreme wealth concentration:
- Restricted Stock Market Participation: Stock market participation is heavily concentrated in the upper wealth tiers (Basak and Cuoco 1998; Board 2025). The bottom 50% of the population holds negligible financial equities; their consumption is hand-to-mouth and uncorrelated with stock market swings.
- Aggregation Fallacy: The marginal investor determining asset prices is not the average consumer. Using aggregate per-capita consumption growth drastically understates the consumption volatility and marginal utility fluctuations of actual equity holders (Mankiw and Zeldes 1991; Yogo 2006).
- Capital Share Shifts: Fluctuations in the capital-labor income distribution induce severe capital share risks that standard representative-agent Euler equations completely ignore (Lettau et al. 2019).
Once socioeconomic heterogeneity and structural participation limits are recognized, the “puzzles” dissolve. They exist only as the predictable failure of a closed-system model that assumes away the structural divisions of real-world capitalism.
6 Normative Free Market Conditions
Neoclassical orthodoxy treats market equilibrium as a spontaneous, self-sustaining state of nature. However, a rigorous comparison of foundational market concepts reveals that equilibrium and efficiency cannot exist in an institutional vacuum.
6.1 Conceptual Matrix of Market Conditions
| Dimension | Perfectly Competitive Market (PCM) | Capital Asset Pricing Model (CAPM) | No-Arbitrage Condition (NA) | Efficient Market Hypothesis (EMH) |
|---|---|---|---|---|
| Market Scope | Competitive physical goods | Competitive capital assets | Fundamental structural boundary | Informational processing state |
| Price Determination | Marginal cost = Marginal utility | Systematic risk premium (\(\beta\)) | Elimination of free lunches | Prices fully reflect information |
| Information Structure | Complete transparency | Identical joint probability beliefs | Not required; supports pricing cone | Necessary condition for efficiency |
| Friction Assumptions | Zero entry/exit costs | Unconstrained risk-free borrowing | Excludes persistent arbitrage | Bounded by transaction/liquidity costs |
| Systemic Role | Pareto-optimal allocation | Efficient risk allocation | Fundamental stability constraint | Price discovery mechanism |
6.2 The Institutional Reality of No-Arbitrage
The No-Arbitrage condition is frequently described as an automated market clearance mechanism. In reality, persistent arbitrage opportunities routinely arise due to institutional rigidities, capital constraints, and asymmetric market power (Shleifer and Vishny 1997). When unconstrained arbitrageurs accumulate excessive capital, they do not restore equilibrium; they form financial monopolies that distort price discovery and impair systemic risk allocation.
No-Arbitrage and market efficiency are not automatic outcomes of deregulation. They are fragile institutional constructs that require active structural safeguards: * Rigorous disclosure mandates to enforce information symmetry. * Anti-monopoly enforcement to dismantle exclusionary economic moats. * Position limits, leverage caps, and circuit breakers to prevent predatory capital concentration. * Neutral market-making architectures to guarantee public liquidity.
7 The Normative Turn in Financial Economics
Natural sciences describe invariant systems: the gravitational constant does not alter its behavior in response to physical theories. Social sciences, however, govern reflexive, open systems where theories actively reshape institutional realities.
The foundational error of modern finance has been treating its models as descriptive natural laws rather than normative institutional aspirations. When neoclassical models insist that markets are closed, self-correcting, and naturally efficient, they provide ideological cover for structural monopolization, rent extraction, and systemic fragility.
True market efficiency, genuine risk-sharing, and no-arbitrage are not pre-existing natural phenomena. They are normative conditions that society must collectively establish, defend, and continually reform. Rebuilding financial theory requires abandoning the illusion of closed-system optimization and constructing an institutional economics that subordinates capital allocation to democratic accountability, genuine productive innovation, and social welfare.