<aside> đĄ Ideally, the below will be co-signed by a small group of relevant academics.
(So far the main authors are Joe Edelman and Tan Zhi Xuan, with input from Atoosa Kasirzadeh, Ryan Lowe, and Manon Revel.)
</aside>
While academic thought has delivered to us many simplified models of social behavior, the 20th century was dominated by a âcentral axisâ of five social theories: microeconomics, game theory, social choice, mechanism design, and welfare economics. These five were all supported from below by rational choice theory[0], and below that, by an understanding of individual actors with fixed preference profiles or utility functions.
We call these the âcentral axisâ, because they were the principle social theories which generated institutional designs, and which justified them. They also provided backing for the dominant political theories, which were freedom- and fairness-based, again focusing on individual actors and the (free or fair) pursuit of their individual preferences.
Something to note about these central axis theories, is they model agents as âcontext-freeâ. Individuals are supposedly born with their preference profiles, utility functions, or payoff matrices intact, and thereâs no place in the models for shared values, shared norms, shared beliefs, group identities, etc, that shape these individual preferences.[1]
Despite this lack of context (or because of it), we imagine these theories achieved their centrality because they are mathematically expressive and powerful theories, parsimonious, and they sat well with widespread philosophical intuitions. Plus, they were âgood enoughâ for the institution design challenges of their day.
But recently, the field of AI alignment pushes against this state of affairs. We see six reasons for this.
The problem with revealed preference. The need to find objective functions or reward models which can safely be used to train AI brings us right back to the debates in welfare economics (most closely associated with Amartya Sen) about the limits of revealed preference as a measure of benefit. But, in the meantime itâs become clearer that businesses, governments, and other entities have indeed learned to exploit individuals under the guise of serving their preferences, using AI.[2] For instance, social media platforms learned to manipulate user engagement leading to addiction-like behaviors, prioritizing platform growth over individual well-being.
Alignment methods based on explicit values[3], or norms[4], are starting to emerge and showing clear advantages over revealed preference-based approaches. In general thereâs much more of an appetite to overcomes these problems with revealed preference than when these debates happened in the field of development economics.[5]
Socio-technical alignment requires new institutional forms. AI alignment challenges seem to require inventing new forms of governance, and many directions for institutional innovation arenât supported by the central axis. To give but one example, the central axis version of social choice is blind to the most powerful lever in deliberation: inspiration. The best mechanisms should not just accommodate existing preferences; they should allow for the formation and inspiration of new ones, by creating environments where individuals grow in their understanding of what is good.[6] Can mechanism design catch up with ancient Athens?
Recently, mechanisms by Conitzer[7] and Klingefjord et al[8] leave the central axis behind and build on different social theories.
The need to model and preserve âthe social fabricâ. One thing we want powerful AI to be careful about is that shared context: the networks of trust, of values alignment, and of normative cooperation which keep society working. The central axis theories, because they donât model agents as embedded in a social context, mostly pretend this social fabric doesn't exist.[9] This means alignment efforts aimed to preserve or enhance the social fabric are hard to build on these theories!
We already see the results of this playing out in society: recent decades see metrics based on preferences or transactions go up. Our âwealthâ is increasing. But liberals often suspect our capacity for social cognition has declined via misinformation, conspiracies, etc; conservatives suspect weâve suffered a decline in morals and aesthetics; both tend to say our norms and channels for cooperation have eroded. Whether such declines are happening or not, it doesnât seem likely that measures of consumption or engagement/revealed preference would show them. These things are important; they should find a prominent place in our social theories and ideas of welfare.
Cooperating AI agents. Another challenge in alignment is to get a vast ecosystem of AI agents cooperating. This turns out to be a sore point for the central axis theories, and a place where ideas about shared norms or values from outside the central axis have already been imported, to address mismatches between the equilibria predicted by context-free models and those observed among real, cooperating agents.
We can work to make AI agents that cooperate as we do, but to do so we may need to adopt more sophisticated (yet tractable), norm- and value-embedded models of human cooperation.
The challenge of super-wisdom. The central axis assumes agents have fixed preferences, disconnected from one another and from any broader notion of the good**.** This means AIs, individuals and societies cannot collectively aspire to ideals beyond preference satisfaction.
However you understand our social embedding â whether as shared values, norms, or beliefs â to acknowledge it exists seems to suggest kinds of goodness beyond preference satisfaction. The context of shared values suggests moral progress or learning. A context of evolving norms suggest game-theoretic notions of goodness, such as cooperation at higher scales, or across diverse ecosystems. With a context of shared beliefs, thereâs the aspiration to discover higher truths.
Yet, so long as individual preferences are the yardstick of the good, none of these other notions can be admitted, and no one is allowed to know better than anyone else. To make progress, revealed preferences must be reformulated as exogenousâas a function of underlying values (themselves subject to moral learning), plus social norms and strategic considerations.
New tools. Finally, displacing the central axis theories seems newly feasible. Itâs likely that the success of the central axis was partly based on the availability of hard data - behavioral data in the form of votes, purchases, and clicks. This was, for a time, far easier to obtain than intersubjective and qualitative data about shared context.
LLMs make the systematic investigation of shared norms, values, and beliefs much easier, because they can (1) perform qualitative interviews at scale; (2) simulate people; or (3) because their weights crystalize these linguistic and conceptual aspects of our social fabric as a form we can see.
This will let new models of socially embedded agents compete for rigor.
All these forces suggest a shift away from the context-free, central axis theories.
This is a paradigm shift, significantly bigger than âbehavioral economicsâ.
Exciting!
Weâd like to lay out a program for responding to this situation. What would we like to see?
Commensurate reach and integration. We believe there are versions of every component of the central axis based on (socially) embedded agents rather than agents with only individual preferences, utility functions, or payoffs:
Letâs call these theories together the Allied Theories of Embedded Agency (ATEA). Thereâll be a period of divergent experimentation, but ultimately these theories must fit together tightly, just as the current central axis does.
Formal guarantees. While there are alternatives to the central axis models, they tend to be less useful for reasoning about institutions, interventions, and mechanisms.
What makes the central axis so good in this regard is that it provides âformal guaranteesâ, verifiable on paper, about the behavior and outcomes of mechanisms and institutions. Guarantees like pareto-optimality, strategy-proofness, and envy-freeness. Via these formal guarantees, the central axis militates for some designs and against others, so designers have results without needing to know a mechanismâs real-world performance.
We can compare this feature of the central axis against vaguer theories, like those in macroeconomics and Marxism. These lack formal guarantees and, instead, the same theoretical apparatus can often be used to argue either for and against something.
We believe the ATEA will provide new formal guarantees. Some will concern new kinds of goodness a mechanism or institution can have, which havenât been properly conceptualized yet.
Algorithms, institutions, and mechanisms that fit us well (and solve real-world problems). The ultimate test of a social theory is whether systems built or justified by it fit us well. What we mean by this[10] is: any set of social theories implies certain skills and activities on the part of agents. Current mechanism design implies agents who optimize, calculate, strategize, collude, reduce values to numbers, estimate probabilities, etc. Standard theories of intelligent agency â which guide the design of both AI systems and their models of humans â do the same. These activities arenât foreign to human nature, but donât seem central either. So, we have to put in a kind of effort to participate in the institutions that emerged from this view of agency.
Other social theories have similarly poor fits: people arenât natural advocates for class or race interests, nor natural status maximizers, etc. We hope that the ATEA will be different â that norm-following, norm-intuiting, inspiration, and pluralist value-pursuit will be a better account of human nature and rationality than the above, and that therefore algorithms, institutions and mechanisms built on the ATEA will better fit our lives. Ultimately, the proof here is experiential: as new context-aware institutions, algorithms, mechanisms, platforms, and laws develop, weâll see what it feels like to participate in them.
Social change along an unexpected but optimistic path. If all of the above go well, weâll see new political theories and even political rhetoric emerging. New political theories will go beyond freedom and fairness, aiming at notions of the good beyond preference satisfaction, as gestured to in point #5 above. These new political theories will legitimate the reorganization of society to achieve these new notions of the good.
This will require a ragtag band of geniuses, collaborating closely. Youâre reading this because we think your work is exceptional, and you might belong in this academic movement.