Is there any potential negative to using cached latents, besides VRAM use?

[morphinapg]

I don't know entirely how the latents thing works, so I was wondering if there was any kind of random seed involved when generating latents, because I would imagine using a new seed each epoch would result in better training for each image.

But if there's nothing like that involved in the process, then I would imagine the results would be identical, just faster, as long as your VRAM can handle it.

On the current version of the extension I can't even train without caching, and it's slower even when I do, but on the older version of the extension, it's about 15% faster if I use cached latents, so it's worth asking.