There are several tabular MDP algorithms such as MBIE-EB, E3, and Q-learning with bonus. We currently only support the latter. It will be great to include more. These could be useful as black boxes for algorithms which learn latent discrete models. They could also be great for demo.