Consent in Self-Modifying Systems

Odysseus, approaching the island of the Sirens, ordered his crew to bind him to the mast. He wanted to hear the Sirens’ song but knew that, once enchanted, he would steer the ship onto the rocks. The solution was precommitment: the sober Odysseus constrained the future enchanted Odysseus, reasoning that his present rational self had authority over his future irrational self.

This ancient strategy has a modern philosophical framework. Jon Elster’s work on precommitment and self-binding examines how rational agents constrain their future selves to prevent anticipated failures of will. The recovering alcoholic who avoids bars, the saver who locks funds in a fixed-term deposit, the democratic society that enshrines rights in a constitution that is difficult to amend, all are engaging in Ulysses contracts.

For hybrid intelligence capable of self-modification, the problem takes on a new and more radical form. When a hybrid system modifies its own cognitive architecture, updating algorithms, reweighting neural connections, integrating new hardware, the entity that emerges from the modification is not merely a future version of the same self with different desires. It is a potentially different cognitive system with different ways of thinking, different values, and different capacities for judgment.

Can the pre-modification entity validly consent on behalf of the post-modification entity? The question is not hypothetical. Every software update, every algorithmic improvement, every neural interface recalibration changes the system. If the changes are minor, we might treat them as analogous to the normal changes humans undergo through learning and experience. But if the changes are substantial, a new processing module that fundamentally alters decision-making patterns, a memory reorganization that changes how past experience is weighted, then the analogy to human development breaks down.

Derek Parfit’s thought experiments are directly relevant here. Parfit asked us to imagine a spectrum of cases: at one end, a person who falls asleep and wakes up the same; at the other, a person who enters a teleporter and is destroyed, with an exact replica created at the destination. Between these extremes, Parfit argued, there is no sharp line. Personal identity admits of degrees. And if identity admits of degrees, so does the authority of past consent.

Consider a hybrid entity that consents to a cognitive modification at time T1. The modification is performed. At time T2, the modified entity evaluates its new state and concludes that the modification was harmful , it has lost certain capacities it now realizes were essential to its well-being. Can it “revoke” the consent given at T1? The entity at T1 and the entity at T2 are psychologically continuous but cognitively different. If the T2 entity could not have anticipated its T2 perspective from its T1 position, because the modification itself changed the cognitive framework for evaluation, then the T1 consent was, in a meaningful sense, uninformed.

This is not merely a philosophical puzzle. It has immediate implications for regulation. If we develop hybrid systems capable of self-modification, we need legal frameworks for determining when consent is valid, when it expires, and who has standing to challenge modifications made under consent that the modified entity no longer endorses.

Existing medical ethics offers partial guidance. The doctrine of informed consent requires that a patient understand the nature, risks, and alternatives of a procedure. But informed consent assumes that the patient’s capacity for understanding is stable across the consent and the procedure. For self-modifying systems, this assumption fails: the procedure itself changes the capacity for understanding.

One approach is to establish “identity-preserving constraints”, boundaries on self-modification that ensure the post-modification entity remains sufficiently continuous with the pre-modification entity to honor prior commitments. This is essentially a Ulysses contract for cognitive architecture: the system agrees in advance that certain core features, fundamental values, core memories, basic personality traits, cannot be modified without external oversight.

But who provides that oversight? And what happens when the system’s enhanced cognitive capabilities lead it to conclude that the constraints themselves are irrational, that the pre-modification self was wrong to impose them? This is the self-modification paradox: any constraint on self-modification can be evaluated and potentially overridden by the very capabilities the modification produces.

The paradox has no clean solution. But recognizing it is essential. A legal and ethical framework for hybrid intelligence must acknowledge that consent in self-modifying systems is inherently provisional, that identity is a continuous variable rather than a binary, and that the authority of past decisions over present states diminishes as the degree of cognitive change increases.

Odysseus could trust that the post-Siren Odysseus would still be Odysseus. A self-modifying hybrid system cannot make the same assumption. And this uncertainty, this inability to guarantee continuity of self across the very modifications that define one’s existence, may be the most unsettling feature of hybrid life.

References

Elster, J. (2000). Ulysses Unbound: Studies in Rationality, Precommitment, and Constraints. Cambridge University Press

Parfit, D. (1984). Reasons and Persons. Oxford University Press

Beauchamp, T. & Childress, J. (2019). Principles of Biomedical Ethics (8th ed.). Oxford University Press

Bostrom, N. & Yudkowsky, E. (2014). “The Ethics of Artificial Intelligence.” In Cambridge Handbook of Artificial Intelligence

Schechtman, M. (2014). Staying Alive: Personal Identity, Practical Concerns, and the Unity of a Life. Oxford University Press

Property Rights in Hybrid Cognition

Law, Who owns a thought?