<aside> 🌀
The Meaning Alignment Institute is helping coordinate the Better AI and Institutions via Thick Models of Choice field-building effort. Here are 6 areas where we believe accelerated progress would be timely.
</aside>
<aside> 👉
I’ve added some desired papers for each of these clusters here.
https://docs.google.com/spreadsheets/d/1YCeWaLUeSgjpau_TMihezTtxapiGzK44mN-H2p-ArhM/edit?gid=0#gid=0
</aside>
Roughly speaking, each area below suggest a kind of research pipeline: { certain techniques need to be developed → then comes a proof of concept system/mechanism → then a variety of legitimating tests and proofs → followed by a policy advocacy paper }.
I personally believe that in each case, the policy space is extremely limited mainly because of a lack of technical contributions, and that norms-based, values-based, or hybrid moral reasoning and thick choice can address that.
Ideally, to head off the first two areas, we’d publish 12-15 papers in 2025! I do think we’re pretty much the group to do it, but we’d have to coordinate tightly.
CA1 - Manipulative Assistants and Agents. AI assistants can (1) incept foreign values into users; (2) persuade users that actions were beneficial which weren’t; or just (3) act in ways that don’t reflect users’ morality, while seeming to help their human users or represent their interests.
CA2 - Norm-Blind Agents. As AI agents with inadequate moral reasoning replace human agents (lawyers, doctors, financiers) important values and professional norms are abandoned, leading to tragic outcomes in finance, business, and even military applications as AGI spreads.
CA3 - Negotiation Failures Among Black-Box Agents. In general, purely strategic agents face problems like the prisoner’s dilemma and have limited means for overcoming these coordination failures.
CA4 - Economy Increasingly Detached from Human Welfare. The gap between capitalist success / social media engagement and human flourishing has already widened, and an economy with many agents and autonomous corporations is likely to further widen it.
CA5 - Democratic Institutions Can’t Keep Up.
Current democratic and global institutions cannot handle geopolitical conflicts nor respond rapidly enough to AI activities.
Research focus: Nimble & Legitimate Democratic Agents. New mechanisms use Kantian or values-based moral reasoning to decide things based on public values and norms, and to foster new types of rapid deliberation, so that we can make much more sophisticated institutions that also better represent the people.
Threat
Manipulative Assistants and Agents. AI assistants can (1) incept foreign values into users; (2) persuade users that actions were beneficial which weren’t; or just (3) act in ways that don’t reflect users’ morality, while seeming to help their human users or represent their interests.
Opportunity
Morally-Competent Agents. Assistants and agents understand their users’ values and reason about how they apply to new situations, in a pattern that’s robust against incentives for manipulation.
More info
Threat
Purely Goal-Directed Agents. As AI agents with inadequate moral reasoning replace human agents (lawyers, doctors, financiers) important values and professional norms are abandoned, leading to chaos and harm.