Some fun times with hibernate indeed. Of course, I have to figure out this myself since the hibernate forums are the most useless forums in existence.
Here's the scenario, and how I fixed it.
I have 2 domain objects in a bi-directional ManyToMany relationship (Groups and Roles). Nothing new here.
The problem happens if you add/remove any associations AFTER the cache has been initialized.