The PNG Guide is an eBook based on Greg Roelofs' book, originally published by O'Reilly.



Real-World Comparisons

The only convincing way to demonstrate the compression benefits of one image format over another is to do an actual comparison of the two on a set of real images. The problem is choosing the set of images--what works for one person may not work for another. What I've done here is to gather together results from a number of real-world tests performed over the past few years. Readers can expect to achieve similar results on similar sets of images, but keep in mind that one can always choose a particular set of images for which the results will be dramatically different. I'll explain that remark after we see a few cases.

For starters, let's look at a small, very unscientifically chosen set of images: seven nonanimated GIF images that happened to be the only ones readily available on my machine one fine day in June 1998.

Table 9-2. Seven Non-Animated, Non-Scientifically Selected GIF Images

 Name   GIF Size 
 linux-penguins   38,280 
 linux-tinypeng   1,249 
 linux_bigcrash   298,529 
 linux_lgeorges   20,224 
 linux_rasterman   4,584 
 sun-tinylogo   1,226 
 techweb-scsi-compare   27,660 
 TOTAL   391,752 

The images ranged in size from just over a kilobyte to nearly 300 kilobytes (the exact byte sizes are given in Table 9-2) and in dimension from 32 × 32 to 800 × 600. All but the first and last were interlaced. When converted to PNG with the gif2png utility (Chapter 5, "Applications: Image Converters"), preserving interlacing manually and introducing no new text annotations, things improved somewhat; Table 9-3 shows the preliminary results.

Table 9-3. Same Seven GIF Images After Conversion to PNG

 Name   PNG Size   Change 
 linux-penguins   35,224   -8.0% 
 linux-tinypeng   722   -42.2% 
 linux_bigcrash   283,839   -4.9% 
 linux_lgeorges   20,476   +1.2% 
 linux_rasterman   4,812   +5.0% 
 sun-tinylogo   566   -53.8% 
 techweb-scsi-compare   20,704   -25.1% 
 TOTAL   366,343   -6.5% 

Five of the images shrank when converted to PNG--three of them quite significantly--while two grew. Overall, the set achieved a 6.5% improvement in byte size. Since gif2png uses the standard settings of the libpng reference code,[73] its results may be considered typical of ``good'' PNG encoders. But the owner of a web site will often be willing to spend a little more time up front on compression in return for additional bandwidth savings in the long run. That's where pngcrush (also discussed in Chapter 5, "Applications: Image Converters") comes in. Table 9-4 shows its results; the percentages are again relative to the original GIF file sizes.

Table 9-4. Same Seven GIF Images After PNG Conversion and Optimization

 Name   Optimized 
 PNG Size 
 Change 
 linux-penguins   34,546   -9.8% 
 linux-tinypeng   710   -43.2% 
 linux_bigcrash   282,948   -5.2% 
 linux_lgeorges   19,898   -1.6% 
 linux_rasterman   4,731   +3.2% 
 sun-tinylogo   550   -55.1% 
 techweb-scsi-compare   19,155   -30.7% 
 TOTAL   362,538   -7.5% 

[73] libpng is discussed at length in Chapter 13, "Reading PNG Images", Chapter 14, "Reading PNG Images Progressively" and Chapter 15, "Writing PNG Images", which demonstrate how to use libpng to read and write PNG images.

So we see that the current state-of-the-art PNG encoder ekes out another percentage point in the total size, with all but one of the images now smaller than the original. That lone holdout is worth a closer look in this case. I already noted that linux_rasterman.gif was one of the interlaced images; suppose it had not been? The noninterlaced GIF version is 4,568 bytes, only 16 bytes smaller than the original. But the noninterlaced PNG version is either 4,067 bytes (gif2png) or 4,000 bytes (pngcrush)--a savings of 11.0% or 12.4% over the noninterlaced GIF. In other words, PNG's two-dimensional interlacing scheme can have a significant negative impact on compression, particularly for small images. This is an important point to consider when creating images: is interlacing really needed for a 152 × 96 image (as in this case) when the penalty is more than 18% of the file size?

This example may have been instructive, but seven images do not constitute a statistically valid sample.[74] So let's consider a few more data sets. One real-life example comes from the course entitled ``Authoring Compelling and Efficient VRML 2.0 Worlds'' at the VRML98 conference in Monterey, California. Though the content of the course was otherwise outstanding, one slide comparing image formats for 3D textures was rather alarming from a PNG perspective. It showed the information displayed in Table 9-5, together with the textures themselves (which are omitted here):

Table 9-5. Original PNG, GIF, and JPEG Comparison from VRML98 Course

Name Dimensions Type JPEG Size GIF Size PNG Size
 linoleum1  128 × 128 grayscale 10,956 7,055 16,008
 doggie  128 × 256 color 9,897 24,605 89,022
 fog  128 × 128 grayscale + alpha -- -- 26,732
 circlefade  128 × 128 grayscale + alpha -- -- 15,735
 buttfly  128 × 128 color + transparency -- 4,367 --

[74] That would be a small understatement.

Even with no more details than are shown here, at least one problem is apparent: in row 2, the JPEG image is 24 bits deep, while the GIF is only 8 bits. Judging by the size of the corresponding PNG, one might assume (correctly) that the PNG is also 24 bits. So on the one hand, PNG is being compared with an image of the same depth that uses lossy compression, while on the other it is being compared with an image only one-third as deep, which also amounts to lossy compression. That hurts.

As it turned out, there were other problems: the PNG images were created with an image editor not known for its compression capabilities, and some of the PNGs were interlaced even though their GIF counterparts were not. (And since this was a VRML course, I should note that no VRML browser in existence actually uses interlacing to render textures progressively, so there is generally no point in creating such images.) The upshot is that all of these factors--JPEG's lossy compression, mixing 24-bit and 8-bit images, mixing interlaced and noninterlaced images, and using a particularly poor encoder to compress the PNGs--worked against our favorite image format.

After evening the playing field by using the GIFs as the source images for the PNGs, turning off interlacing, and using a combination of conversion and encoding tools (including pngcrush), the results were considerably better for PNG, as shown in Table 9-6.

Table 9-6. Updated PNG, GIF, and JPEG Comparison for VRML98 Course Images

Name JPEG Size GIF Size Original
PNG Size
Optimized
PNG Size
PNG
Change
linoleum1 10,956 7,055 16,008 6,753 -57.8%
doggie 9,897 24,605 89,022 22,555 -74.7%
fog -- -- 26,732 16,221 -39.3%
circlefade -- -- 15,735 6,638 -57.8%
buttfly -- 4,367 -- 3,965 --

Here, I've marked the smallest version of each image in boldface type; the only one that isn't a PNG is the color JPEG, which is hardly surprising. What is interesting is that the grayscale JPEG (linoleum1.jpg) is larger than both the GIF and optimized PNG versions, despite the presumed benefits of lossy compression. There are at least three reasons for this. First, GIF and PNG both get an automatic factor-of-three savings from the fact that each pixel is only 1 byte deep instead of 3 bytes. Second, JPEG is at a relative disadvantage when dealing with grayscale images, because most of its compression benefits arise from how it treats the color components of an image. Third, this particular image is more artificial than natural, with quite a few relatively sharp features, which makes it particularly ill suited to JPEG-style compression.

But perhaps the most striking feature of Table 9-6 is just how poorly the original encoder did on its PNG images. Realizable savings of 40% to 75% are unusual, but thanks to poor encoding software, they are not as unusual as one might hope.

As another real example (but one that is perhaps more representative of what a typical web site might expect), the owner of http://www.feynman.com/ found that when he converted 54 nonanimated GIFs to PNGs, the collection grew in size from 270,431 bytes to 327,590 bytes. Insofar as all of the original images had depths of 8 bits or less--and even the worst PNG encoder will, on average, do as well or better than GIF on colormapped PNG images--the most likely explanation for the 21% increase in size is that the conversion utility produced 24-bit RGB PNGs. Indeed, the owner indicated that he had used ImageMagick's convert utility, older versions of which reportedly had the unfortunate habit of creating 24-bit PNGs unless explicitly given the -depth 8 option. (This problem seems to have been fixed in more recent versions, but even current versions include 160 bytes' worth of text and background chunks per image.) When the original GIFs were converted to PNG with gif2png instead, the total size dropped to 215,668 bytes, for a 20% overall savings over GIF. Individually, the GIFs were smaller in 15 of the 54 cases, but never by more than 340 bytes. Of the 39 images in which the PNG version was smaller, one-third of them differed by more than a kilobyte, and one was 14 KB smaller.

For the last GIF comparison, I downloaded the World Wide Web Consortium's icon collection, consisting of 448 noncorrupted GIF images. Of these, 43 had embedded text comments and 39 were interlaced. Most of the images were icon-sized (as would be expected), at 64 × 64 or smaller, but there were a handful of larger images, too. The total size of the files was 1,810,239 bytes. Conversion to PNG via gif2png, handling the interlaced and noninterlaced images separately in order to preserve their status, resulted in a total PNG size of 1,587,337 bytes, or a 12.3% reduction. Additional compression via pngcrush resulted in a total of 1,554,965 bytes, or a 14.1% reduction (relative to the GIF size). Out of the 448 images, PNG won the size comparison in 285 cases, lost in 161 cases, and tied in 2 cases. As in the previous test, however, the magnitude of the differences was the critical factor: GIF won by more than a kilobyte in only 1 case, while PNG won by that amount in 37 cases--4 of which were greater than 10 KB, 1 more than 64 KB. The average difference for the 285 cases in which PNG was smaller was 940 bytes; for the 161 GIF cases, it was a mere 78 bytes.

Finally, I've mentioned an upcoming JPEG standard for lossless compression a couple of times; it's worth a quick look, too. JPEG-LS, as the standard will be known,[75] is based on Hewlett-Packard's LOCO-I algorithm. As this is written, it is implemented in version 0.90 of HP's locoe encoder, available only in binary form for the HP-UX, Solaris, IRIX, and 32-bit Windows platforms. (An independent implementation is available as C source code from the University of British Columbia.) In a comparison performed by Adam Costello, the HP encoder was tested against pnmtopng and pngcrush on the eight standard color images in the Waterloo BragZone's ColorSet. pnmtopng is of interest only for speed reasons; even though it is moderately fast, locoe was considerably faster. I have omitted its size results from the comparison since, as expected, pngcrush outperformed it in all cases, though at a considerable cost in speed.

[75] In December 1998 it became an ISO Draft International Standard, the final voting stage before becoming a full International Standard. It will officially be known as ISO/IEC 14495-1 upon approval. It has already been approved as International Telecommunication Union (ITU) Recommendation T.87.

The results were fascinating. In the five test images categorized by the University of Waterloo as ``natural,'' JPEG-LS beat PNG by between 5% and 11%--not a huge difference, but certainly significant. However, in the three images marked ``artistic,'' PNG proved superior by wide margins, with one image more than three times smaller than the corresponding JPEG-LS version. These results are summarized in Table 9-7; once again, the byte size of the winning format for each image is highlighted in boldface type.

Table 9-7. PNG and JPEG-LS Comparison for Waterloo BragZone Color Images

Classification Name Total
Pixels
JPEG-LS
Size
PNG
IDAT Size
Relative
Difference
``natural'' lena 262,144 445,799 475,430 +6.6%
monarch 393,216 555,012 615,260 +10.9%
peppers 262,144 385,047 425,560 +10.5%
sail 393,216 767,374 808,606 +5.4%
tulips 393,216 616,536 680,881 +10.4%
``artistic'' clegg 716,320 653,299 484,589 -25.8%
frymire 1,235,390 935,285 251,865 -73.1%
serrano 499,426 293,532 106,765 -63.6%

Note that in the final column I used the JPEG-LS size as the reference, which effectively works against PNG--had I used PNG instead, the frymire image, for example, would show JPEG-LS as 271.3% larger, which looks much more impressive! Also note that I used the size of the PNG IDAT data for comparison rather than the actual PNG file size; this was done because locoe appears to encode raw JPEG data, with none of the overhead of standard JPEG file formats like JFIF and SPIFF.

In any case, the results are only slightly more statistically valid than the first comparison of GIF images was. Eight samples, even if they are a carefully chosen set of standard research images, cannot tell the full story. And results as intriguing as these certainly deserve more extensive testing, which will no doubt happen in due course.




Last Update: 2010-Nov-26