In Adobe® Illustrator® and similar applications, you can use a group of objects as a shape through which other objects will show. Such a group is called an opacity mask. Conceptually speaking, the objects in the opacity mask are converted to grayscale. Wherever the opacity mask is white, the other objects will be fully visible. When the opacity mask is black, the other objects are hidden.
In a PDF written by Adobe® Illustrator® and similar applications, opacity masks are defined in PDF constructs called soft masks. Any group of objects can be used as an opacity mask. These objects can be ordinary CMYK objects, but they can also be spot color objects.
The use of objects using spot colors in a soft mask can lead to unpredictable results, i.e., the output on a RIP can be different from what you see in Adobe® Acrobat®.
How is that possible? What can you do to avoid this problem?
There is no "correct" handling of soft masks
The root cause of this problem is that, contrary to what one would expect, the algorithms used for transparency calculations using PDF soft masks are not defined. The PDF specification (ISO 32000-1) says the following, in case the transparency group defining the soft mask (e.g.: the objects in the Adobe® Illustrator® opacity mask) uses a device color space:
"For device colour spaces, convert the color to DeviceGray by implementation-defined means and use the resulting gray value as the luminosity, with no compensation for gamma or other colour calibration".
In other words: according to the ISO specification, there is no right and no wrong implementation. Any vendor, implementing a PDF compliant system, is free to implement transparency using soft masks any way it likes.
Soft mask handling in Adobe applications
Adobe® applications, including Acrobat®, consistently use the same method. However, this is an undocumented Adobe® proprietary implementation. Although the expectation in the industry is that all applications should behave in the same way as Acrobat®, this is by no means a more correct implementation than any other method. Since the method used by Adobe® is not documented, it is also not possible to guarantee that the implementations are 100% compatible with the Adobe® methodology.
Soft mask handling in Esko applications
By means of reverse engineering, Esko has implemented an algorithm that is compatible with Acrobat® for soft masks that contain CMYK objects.
The task is however, a bit more challenging when the soft mask objects contain spot colors. The spot colors first need to be converted to CMYK before they can participate in the soft mask calculations. Again there are different possible algorithms to convert a spot color to CMYK, all of them equally correct but all leading to different results.
Up till recently, Esko applications used the Esko Color Engine for conversion of a spot color to CMYK under all circumstances. This is different (but equally correct) from the algorithm used in Acrobat®.
In Suite 14 (or through a HF on Suite 12 Assembly 4 and Assembly 5), we have changed the Esko algorithm to be in line with Adobe®'s algorithm in case the alternate color space representation of the spot color is CMYK. This will be the case, if you set the color mode of a spot color to CMYK.in Adobe® Illustrator® (this was the default prior to Creative® Suite® 6).
In Suite14, assembly 7 (or through a hotfix on 14.1), we further refined the behaviour into : when normalizing an existing non-normalized PDF, the CMYK values of spot colors are determined as follows
when a spot color is used outside an opacity mask, then the Esko application uses the CMYK values from the Esko Color Engine, in order to ensure correct color proofing
when a spot color is used inside an opacity mask, then the original CMYK values from the PDF are used , in order to preserve the correct opacity
when a spot color is used both inside and outside an opacity mask, then a second spot color is created (with an underscore prefix), for usage inside the opacity mask, using the original CMYK values from the PDF
When the alternate color space of the spot color is Lab (this is the default since Creative® Suite® 6), the results will still differ between what you see in Acrobat® and what you get in an Esko application.
Again, according to ISO specifications, this difference does not mean that the Esko implementation is wrong.