You can check your log to ensure the cache is being read. If Yuzu ignores the file, it might be corrupted or for the wrong version.
You can download "complete" shader caches from community forums like r/128bitbay to skip the stuttering phase entirely. How to Install a Shader Cache Zelda Totk Shader Cache Yuzu-
The Depths (the underground area) uses a unique set of shadow and fog shaders. Many incomplete caches ignore the Depths. Fix: You need a "Depths-complete" cache. Search for a cache marked v1.2.0 (the game update) with Depths verified . If you cannot find one, manually build the cache by entering the Depths, gliding around for 10 minutes, then exiting. Your cache will then be stable. You can check your log to ensure the cache is being read