Transformer Metacontroller
Project description
metacontroller
Implementation of the MetaController proposed in Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning, from the Paradigms of Intelligence team at Google
Install
$ pip install metacontroller-pytorch
Appreciation
-
Pranoy for submitting a pull request for fixing the previous latent action not being included in the inputs to the switching unit
-
Diego Calanzone for proposing testing on BabyAI gridworld task, and submitting the pull request for behavior cloning and discovery phase training for it!
-
Andrew Song for ongoing implementation of the PinPad environment!
-
Diego Calanzone for his experimental acumen, bringing the project to an initial working state for the BabyAI environment!
-
Andrew Song for implementing linear probing and fixing an issue with the action space
-
Andrew Song for identifying a critical issue with past action embed handling and detaching gradients of target states
-
Diego Calanzone for identifying inconsistencies in the MetaController
Usage
import torch
from metacontroller import Transformer, MetaController
# 1. initialize model
model = Transformer(
dim = 512,
action_embed_readout = dict(num_discrete = 4),
state_embed_readout = dict(num_continuous = 384),
lower_body = dict(depth = 2),
upper_body = dict(depth = 2)
)
state = torch.randn(2, 128, 384)
actions = torch.randint(0, 4, (2, 128))
# 2. behavioral cloning (BC)
state_loss, action_loss = model(state, actions)
(state_loss + action_loss).backward()
# 3. discovery phase
meta_controller = MetaController(
dim_model = 512,
dim_meta_controller = 256,
dim_latent = 128
)
state_pred_loss, action_recon_loss, kl_loss, aux_ratio_loss = model(
state,
actions,
meta_controller = meta_controller,
discovery_phase = True
)
# they did not use state pred loss in the paper (weight set to 0, but available)
# the ratio loss from h-net paper is also available, but optional (set ratio_loss_weight > 0)
(action_recon_loss + kl_loss * 0.1).backward()
# 4. internal rl phase (GRPO)
# ... collect trajectories ...
logits, cache = model(
one_state,
past_action_id,
meta_controller = meta_controller,
return_cache = True
)
meta_output = cache.prev_hiddens.meta_controller
old_log_probs = meta_controller.log_prob(meta_output.action_dist, meta_output.actions)
# ... calculate advantages ...
# for GRPO, the inputs to policy loss should be of shape (batch, seq, dim_latent)
# where dim_latent is the dimension of the latent action space
loss = meta_controller.policy_loss(
group_states,
group_old_log_probs,
group_latent_actions,
group_advantages,
group_switch_betas
)
loss.backward()
Or using evolutionary strategies for the last portion
# 5. evolve (ES over GRPO)
model.meta_controller = meta_controller
def environment_callable(model):
# return a fitness score
return 1.0
model.evolve(
num_generations = 10,
environment = environment_callable
)
Citations
@misc{kobayashi2025emergenttemporalabstractionsautoregressive,
title = {Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning},
author = {Seijin Kobayashi and Yanick Schimpf and Maximilian Schlegel and Angelika Steger and Maciej Wolczyk and Johannes von Oswald and Nino Scherrer and Kaitlin Maile and Guillaume Lajoie and Blake A. Richards and Rif A. Saurous and James Manyika and Blaise Agüera y Arcas and Alexander Meulemans and João Sacramento},
year = {2025},
eprint = {2512.20605},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2512.20605},
}
@article{Wagenmaker2025SteeringYD,
title = {Steering Your Diffusion Policy with Latent Space Reinforcement Learning},
author = {Andrew Wagenmaker and Mitsuhiko Nakamoto and Yunchu Zhang and Seohong Park and Waleed Yagoub and Anusha Nagabandi and Abhishek Gupta and Sergey Levine},
journal = {ArXiv},
year = {2025},
volume = {abs/2506.15799},
url = {https://api.semanticscholar.org/CorpusID:279464702}
}
@misc{hwang2025dynamicchunkingendtoendhierarchical,
title = {Dynamic Chunking for End-to-End Hierarchical Sequence Modeling},
author = {Sukjun Hwang and Brandon Wang and Albert Gu},
year = {2025},
eprint = {2507.07955},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2507.07955},
}
@misc{fleuret2025freetransformer,
title = {The Free Transformer},
author = {François Fleuret},
year = {2025},
eprint = {2510.17558},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2510.17558},
}
Life can only be understood backwards; but it must be lived forwards - Søren Kierkegaard
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file metacontroller_pytorch-0.2.0.tar.gz.
File metadata
- Download URL: metacontroller_pytorch-0.2.0.tar.gz
- Upload date:
- Size: 36.9 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.9.25
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8cce07bfc6d755fb624fabaf64095cbe2d6827ccd40b323a9b132b02df99a910
|
|
| MD5 |
46dd783210a90f82485cb1e6f24230a6
|
|
| BLAKE2b-256 |
6cd882fcba143eb432ddd050fb6077c42db7a67ba0ff0fa7c051250bf30f865d
|
File details
Details for the file metacontroller_pytorch-0.2.0-py3-none-any.whl.
File metadata
- Download URL: metacontroller_pytorch-0.2.0-py3-none-any.whl
- Upload date:
- Size: 18.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.9.25
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a41761273f11246f093580137f9f6607906d4e86672ff08e0abb93f2498df019
|
|
| MD5 |
de61d40e73bd714b0adaed39eb14c308
|
|
| BLAKE2b-256 |
778042bdd2e58b6faea1278ed0b6f489630de848727f62d5b7def6df77a2ab3f
|
