Marble Refracion Tests

What I'm trying to accomplish is a marble with a refractive glass effect,
with the colorful "swirl" visible in the middle.

This is difficult because the Three.js refraction effect is not true refraction, it uses environment mapping.

Test #1

As in this first experiment, it only reacts to the environment cube map, and it does not show any geometry inside or behind the refractive sphere.
Refraction Test #1
You can rotate the camera, and adjust the refraction index.
Change the opacity to see the "swirl" placeholder. (Make sure "transparent" is checked first)

Test #2

Refraction Test #2
(Upgraded with Three.js Orbit Controls - drag mouse to move camera)

This time, I tried to separate the marble shell into 2 parts, front and rear.
These two halves are locked to the camera, so that the rear half is always concave toward the camera,
and the front surface is always convex toward the camera.
The inside of the rear surface will render the refraction effect, so that it is properly occluded behind the "swirl".

The front surface will be mostly transparent, in order to show the swirl in the middle,
and it will handle some reflection/Phong lighting effects.

The problem right now is that the refraction is being rendered on the rear spere, and it is appearing right-side-up.
However, as with a spherical lens, the refracted light should appear upside-down.

I've tried flipping the texture UV values, but it did not seem to have any effect.
My idea is to try to render the refraction effect for the front sphere surface, but then draw it onto the rear half of the sphere.