Overprinting is a PDF feature since version 1.2 of the Reference Manual and has evolved since then. The ISO 32'000 standard clearly defines overprinting for devices that support the feature, especially in conjunction with patterns, shadings and transparency. It doesn't say, however, anything about how to compute a resulting color in a preview function. This is somehow unfortunate, since some of the sub-standards, such as PDF/A, claim the fact that a unique reproduction is guaranteed.
Anyway. The result of overprinting is highly device dependent. So, a preview function that simulates overprinting can only approximate the resulting color appearance. Due to the lack of a specification, as always in such cases, developers are trying to refer to the Adobe Acrobat implementation. However, the method Acrobat uses is not publicly documented.
After bringing up the topic in an ISO working group I got hold of the functional diagram. The overprint preview function is surprisingly simple and easy to implement. It is similar to a blend function in the transparency model but works differently.
The overprint preview function is applied as a last step. The rendering engine renders the content of a page on a bitmap surface with separate color channels. Each of these color channels describe the color intensity of a colorant. The colorant belongs to the process colors (CMYK) or to spot colors (PANTONE etc.) used to create the page. The overprint preview function now transforms each pixel of the target surface to a pixel of the preview surface (RGB) by blending each pixel using the said blend function.
I have implemented the method in our own rendering engine and the result looks similar to Acrobat. If you have your own experience with implementing overprinting preview or have any questions please let me know.