Archive for the 'images' Category

Uncompressed data in base64? Probably not

Thursday, February 4th, 2010

The beauty of experimentation is that failures are just as fun as successes. Warning: this post is about a failure, so you can skip it altogether :)

The perf advent calendar was my attempt to flush out a bunch of stuff, tools and experiments I was doing but never had the time to talk about. I guess 24 days were not enough. Here's another little experiment I made some time ago and forgot about. Let me share before it disappears to nothing with the next computer crash.

I've talked before about base64-encoded data URIs. I mentioned that according to my tests base64 encoding adds on average 33% to the file size, but gzipping brings it back, sometimes to less than the original.

Then I saw a comment somewhere (reddit? hackernews?) that the content before base64-encoding better be uncompressed, because it will be gzipped better after that. It made sense, so I had to test.

"Whoa, back it up... beep, beep, beep" (G. Constanza)

When using data URIs you essentially do this:

  1. take a PNG (which contains compressed data),
  2. base64 encode it
  3. shove it into a CSS
  4. serve the resulting CSS gzipped (compressed)

See how it goes: compress - encode - compress again. Compressing already compressed data doesn't sound like a good idea, so it sounds believable that skipping the first compression might give better results. Turns out it's not exactly the case.

Uncompressed PNG?

The PNG format contains information in "chunks". At the very least there's header (IHDR), data (IDAT) and end (IEND) chunks. There could be other chunks such as transparency, background and so on, but these three are required. The IDAT data chunk is compressed to save space, but it looks like it doesn't have to be.

PNGOut has an option to save uncompressed data, like
$ pngout -s4 -force file.png

This is what I tried - took several compressed PNGs, uncompressed them (with PNGOut's -s4), then encoded both with base64 encoding, put them in CSS, gzip the CSS and compared file sizes.

Code

<?php
// images to work with
$images = array(
  'html.png',
  'at.png',
  'app.png',
  'engaged.png',
  'button.png',
  'pivot.png'
);
//$images[] = 'sprt.png';
//$images[] = 'goog.png';
//$images[] = 'amzn.png';
//$images[] = 'wiki.png';

// css strings to write to files
$css1 = "";
$css2 = "";

foreach ($images as $i) {

  // create a "d" file, d as in decompressed
  copy($i, "d$i");
  $cmd = "pngout -s4 -force d$i";
  exec($cmd);

  // selector
  $sel = str_replace('.png', '', $i);

  // append new base64'd image 
  $file1 = base64_encode(file_get_contents($i));
  $css1 .= ".$sel {background-image: url('data:image/png;base64,$file1');}\n";
  $file2 = base64_encode(file_get_contents("d$i"));
  $css2 .= ".$sel {background-image: url('data:image/png;base64,$file2');}\n";

}

// write and gzip files
file_put_contents('css1.css', $css1);
file_put_contents('css2.css', $css2);
exec('gzip -9 css1.css');
exec('gzip -9 css2.css');

?>

Results

I tried to keep the test reasonable and used real life images - first the images that use base64 encoding in Yahoo! Search results. Then kept adding more files to grow the size of the result CSS - added Y!Search sprite, Google sprite, Amazon sprite and Wikipedia logo.

test with compressed PNG, bytes with uncompressed PNG, bytes difference, %
Y!Search images 700 1506 54%
previous + Y!Search sprite 5118 8110 36%
previous + Google sprite 27168 40836 33%
previous + Amazon sprite + Wikipedia logo 55804 79647 29%

Clearly starting with compressed images is better. Looks like the difference becomes smaller as the file sizes increase, it's possible that for very big files starting with uncompressed image could be better, but shoving more than 50K of images inline into a CSS file seems to be missing the idea of data URIs. I believe the idea is to use data URIs (instead of sprites) for small decoration images. If an image is over 50K it better be a separate request and cached, otherwise s small CSS tweak will invalidate the cached images.

 

One-click Minifier Gadget (OMG) - initial checkin

Sunday, January 31st, 2010

So I've been thinking and talking to folks about this idea of having one-stop shop for all your minification needs. Minification of JS and CSS as well as image optimization helps site performance by reducing download sizes. This is good. But not a lot of people do it.

People don't do it, because it's a PITA :) It's simple enough, but with deadlines upon you and all that, you don't want to do an extra step. That's why having a build process helps, by automating this. But setting up a build process is yet another PITA. So it goes.

So my idea was to help busy designers and developers, that wouldn't invest their time researching which minifiers are good, downloading setting up, learning about the 10+ PNG optimization tools... That's how the the idea for the one-click OMG tool came about. (One-drag is more appropriate, come to think of it...) One tool that runs on all operating systems - Win, Mac, Linux - and delivers all minification and optimization tools you need as one package.

Running

Running the tool is as simple as drag/dropping a bunch of files and directories. Here I've dropped "wordpress" directory. The tool recursively looks into the dropped files for things it can optimize. More information here.

OMG screenshot

Download

Version 0.0.1 is here. It doesn't do image optimization, only JS and CSS minification, but please feel free to download and give it a shot. Unzip the package for your OS and run omg.exe (Windows), OMG.app (Mac), or the omg binary (Linux)

Open source

The code is on GitHub. Fork and enjoy.

The developer's notes are there too - how to setup, run, package. Also a list of todos if you want to help.

Next?

This is just a preliminary version. Feel free to join, comment, suggest. Hate the name? Say so :)

Personally, looks like my plate is very full for the next moth or two, so I probably won't be actively working on the tool. I hope though the foundation is good enough and relatively documented, should be easy to pick up if anyone's interested in contributing.

Built with XUL

This has been a learning experience for me with XULRunner. I loved it. I love the idea of being able to create cross-OS desktop apps with JavaScript alone.

Behind the scenes, I'm using my JavaScript port of YUICompressor's CSSmin and Doug Crockford's JSMin. JSMin should be replaced with YUICompressor (or Google closure compiler) in the next release.

 

CSS performance: UI with fewer images

Wednesday, December 23rd, 2009

Dec 23 This post is the one-before-last article in the 2009 performance advent calendar experiment.

Often performance improvements come with their drawbacks, sometimes improving performance causes pains in other parts of the development process or strips stuff from the final product. Sometimes there's even a conflict where you have to pick: slow, unusable and beautiful or fast and looking like hacked with a blunt axe. But it doesn't have to be this way.

This post outlines some approaches to achieving common UI elements using CSS tricks in as many browsers as possible while using as fewer images as possible. Some of the tricks are brand new, some are very, very old, IE5.5. old. They all have in common the "fewer or no images" mantra. Using fewer images comes with some pronounced benefits:

  • less time spent in Photoshop
  • lighter page, less HTTP requests, less image payload
  • fewer elements in the sprite to maintain (and sometimes fewer sprites) which means longer lived sprites with fewer updates and cache invalidation
  • generally easier maintenance - it's much easier to change a color value than to update and push a new image version

Sometimes some browsers may not be fully supported but that's ok - as long as there's progressive enhancement and the basic page is usable, people rarely notice 1px glows and other ornaments.

So let's get started. BTW, a test page with the stuff discussed in the post is here.

Rounded corners

Yep, let's tackle the biggie.

Forget rounded corners in browser that don't support border-radius. Period. It may be hard to argue this case, but definitely try. Doing rounded corners any other way than border-radius is a pain - it adds markup bloat, it makes you create more images or sprite elements. It's tougher to update. Just forget it. Forget rounded corners in IE < 9 (as rumor has it border-radius is coming to IE9). People may argue that IE is important for your audience. No doubt that's true, but rounded corners are not so important for the audience. Show your designer Yahoo Search results page - the sidebar on the left-hand side. Not very rounded in IE. Do you think this was an easy battle - losing rounded corners in IE for such a high-profile site? Ask the man who won the battle ;)

So starting with a normal module - head, body and border:

The markup - nice and clean:

  <div class="module">
    <div class="hd"><h3>This is the header</h3></div>
    <div class="bd">
      <p>Here comes the content</p>
      <p>Here comes some more</p>
      <p>You can never have too much content, because
         content is king, right?
      </p>

    </div>
  </div>

Some fairly simple border radius to support Firefox, Webkit (Safari, Chrome, iPhone...) and, since a few days ago, Opera 10.5 alpha:

.module {
  -moz-border-radius: 9px;
  -webkit-border-radius: 9px;
  border-radius: 9px;
}

Result:

This is it! Easy-peasy, lemon squeezy.

Now, it's a little annoying to write three declarations for the same thing, but, hey - beats images and extra markup hands down. Also annoying are the differences when setting individual corners (-moz-border-radius-topleft is -webkit-border-top-left-radius). In this case we need to also round the header (class .hd) so it doesn't bleed through the beautifully rounded corners:

.hd {
  -moz-border-radius: 8px 8px 0 0;
  -webkit-border-top-left-radius: 8px;
  -webkit-border-top-right-radius: 8px;
  border-radius: 8px 8px 0 0;
}

Verdict:

  • Full support: Firefox, Safari, Chrome, Opera 10.5
  • Fallbacks: IE (corners are not rounded)

Drop shadows and glows

Another favorite effect designers love - dropping shadows. It's easy to enhance that existing .module without any new images:

.module {
  /* offset left, top, thickness, color with alpha */
  -webkit-box-shadow: 5px 5px 5px rgba(0, 0, 0, 0.5);
  -moz-box-shadow: 5px 5px 5px rgba(0, 0, 0, 0.5);
  box-shadow: 5px 5px 5px rgba(0, 0, 0, 0.5);
  /* IE */
  filter:progid:DXImageTransform.Microsoft.dropshadow(OffX=5, OffY=5, Color='gray');
  /* slightly different syntax for IE8 */
  -ms-filter:"progid:DXImageTransform.Microsoft.dropshadow(OffX=5, OffY=5, Color='gray')";
}

And now our module casts a shadow:

Now two notes for IE: first the shadow doesn't have alpha so it's not as nice and second, this filter may not play along with other filters in the same module. But the shadow is cast and that's a check for IE too, even IE5.5!

You may notice that in this case we basically need to more or less repeat the same declaration three times and the IE declaration two times. This is irksome, but hopefully keeping the strings close together should help gzip compression.

As for glowing, it's the same thing in FF, Webkit, Opera, only without any offset. For IE, there's a different filter called glow:

.glow {
  -webkit-box-shadow: 0 0 10px rgba(50, 50, 50, 0.8);
  -moz-box-shadow: 0 0 10px rgba(50, 50, 50, 0.8);
  box-shadow: 0 0 10px rgba(50, 50, 50, 0.5);
  filter:progid:DXImageTransform.Microsoft.glow(Strength=5, Color='gray');
  -ms-filter:"progid:DXImageTransform.Microsoft.glow(Strength=3, Color='gray')";
}

I added these declaration to a new class .glow so I can add the class name to modules that need to glow. The result:

The result as it glows in IE:

Now you see why I added only 3 pixels glow in IE and whole 5 in the rest. The IE glow is a little .. interesting. Also in IE8 (could be my VM, in IE6 XP no VM all looks OK) the glow seems to move slightly when you hover over the module.

Verdict for shadows and glows:

  • Full support: FF, Safari, Chrome, Opera, IE5.5 and up

More info:

Gradients

Ah, gradients. Sometimes so subtle that we, muggles and other mere mortals, don't see them even when we try our hardest. But for the designer they could be life/death situation.

Let's make the head (class .hd) of our module a gradient without any images:

.hd {background-image: -moz-linear-gradient(top, #641d1c, #f00);
  background-image: -webkit-gradient(linear, left top, left bottom, from(#641d1c), to(#f00));
  filter:progid:DXImageTransform.Microsoft.gradient(startColorstr=#ff641d1c,endColorstr=#ffff0000);
  -ms-filter: "progid:DXImageTransform.Microsoft.gradient(startColorstr=#ff641d1c,endColorstr=#ffff0000)";
}

The result:

What a beautiful (code-speaking, of course, not so sure about visually beautiful) module. It has rounded corners, drop shadows and a gradient and so far we haven't used even a single image. Which means this reddish module can become blue, green or, god forbid, pink - with a single tweak in the code, the CMS or the user preferences (if you're building a social network for example).

Gradients verdict:

  • Full support: FF, Safari, Chrome, IE
  • Fallbacks: Opera (solid color)

More info:

... and RGBA for all

Being able to set the transparency of the background without affecting the transparency of the foreground (the text) is quite handy. That's why there's rgba() in CSS (red, green, blue, alpha). IE is not yet supporting it, but we can use the gradient filter which does support transparency. In this case we don't need the actual gradient so we set start and end color to the same thing. Also the background: transparent is needed for the whole thing to work in IE:

.rgba {
  background-color: transparent;
  background-color: rgba(200,200,200,0.8);
  filter:progid:DXImageTransform.Microsoft.gradient(startColorstr=#99dddddd,endColorstr=#99dddddd);
  -ms-filter: "progid:DXImageTransform.Microsoft.gradient(startColorstr=#99dddddd,endColorstr=#99dddddd)";
}

The result is pleasantly cross-browser:

RGBA verdict

  • Full support: Firefox, Safari, Opera, Chrome, IE

Rotating images

It happens that sometimes you use the same image only flipped. For example open/close thingies, menus and such. How about reusing the same image and rotating it with CSS?

.arrow {background: url(arrow.png) no-repeat; display: block; float: left; width: 33px; height: 33px;}
.right{ /* this is the original image*/ }
.left {
  -moz-transform: rotate(180deg);-webkit-transform: rotate(180deg); -o-transform: rotate(180deg);
  filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=2);
  -ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=2)";}
.up {
  -moz-transform: rotate(270deg);-webkit-transform: rotate(270deg); -o-transform: rotate(270deg);
  filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=3);
  -ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=3)";}
.down {
  -moz-transform: rotate(90deg);-webkit-transform: rotate(90deg); -o-transform: rotate(90deg);
  filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=1);
  -ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=1)";
}

Here's the result. Single image:

Arrow

Result:

the HTML:

<span class="arrow right"></span>
<span class="arrow left"></span>
<span class="arrow up"></span>
<span class="arrow down"></span>

You may notice that the CSS could be quite verbose for saving such small images. It's highly recommended you add the rotation code to a class and use the class name when necessary instead of repeating the same long declaration for every use case or image. Then pray to the gods of compression that this thing gzips well ;)

Verdict

  • Full support: Firefox, IE, Safari, Opera, Chrome

Multiple UI elements with the same background image

The last few tricks have something in common - they each use one background image. The background images are very small - usually around 100 bytes. The tiny image has some transparency to it and is placed as a background-image which sits on top of a background-color. Because of the transparency, the background color shines through, but differently depending on the transparency level of the image above it.

The result is - different UI elements with different colors (and even different hover colors) which can be part of CMS or part of user's skinning and they all reuse the same tiny background. So what can we do this way? A lot of interesting background effects, but here's a few.

Glossy buttons

Here's the end result:

All these buttons share the same background image. The image is 1x1000 and repeated horizontally. The 1000 is just to be safe, very safe, because 50, 100 or 1000 doesn't affect the file size which just a mere 100 bytes. The upper half of the image is a little less transparent. The lower half is 100% transparent. When placed on top of the solid color the whole thing looks shiny and glossy. And you can change the color any way you like.

The HTML:

<p class="button">button1<p>
<p class="button button2">button2<p>
<p class="button button3">button3<p>
<p class="button button4">button4<p>
<p class="button button5">button5<p>
<p class="button button6">button6<p>

And the CSS can't be simpler:

.button {
  background-image:url(http://tools.w3clubs.com/mask/mask.php?x=1000&type=h);
  background-position: center;
}
.button:hover {background-color: #F29222;}
.button2 {background-color: #A41D1C;}
.button3 {background-color: #0F6406;}
.button4 {background-color: #333f79;}
.button5 {background-color: black;}
.button6 {background-color: orange; color: black;}

Actually in the test page I have inlined the image with data URI to save the whole HTTP request for such a teeny image.

As you can see in the URL of the background, I've done a little script to generate some background images:
http://tools.w3clubs.com/mask/mask.php?x=1000&type=h
The image generator's source code is right here.

Stripes

Same technique - but used to generate striped background:

It's basically the same code, only using a different call to the image generator to give us a different background image.

HTML:

  <div class="module stripe earth glow">
    <div class="bd">
      <p>striped background</p>
    </div>

  </div>

  <div class="module stripe tech glow">
    <div class="hd phony stripe"><h3>stripity-stripes</h3></div>
    <div class="bd">
      <p>striped background with the same background image</p>
    </div>
  </div>

CSS:

.stripe {background-image: url(http://tools.w3clubs.com/mask/mask.php?type=stripe);}
.earth  {background-color: olive;}
.tech   {background-color: #bbb;}
.phony  {background-color: #0F6406;}

Again, this image can be a data URI so we save the single HTTP request.

And another gradient

So if you don't like the previously discussed way to do gradients, here's another one. The same trick with the solid color background and a semi-transparent image on top.

Result:

The background images as generated by the service are:
// lighter at the top
http://tools.w3clubs.com/mask/mask.php?type=gradient
// darker at the top
http://tools.w3clubs.com/mask/mask.php?type=gradient&flip=1

Again, you can see the test page here and the source for the image generation is here.

For yet another example of this technique check my post on this (abandoned) blog phonydev.com. There I take an image and a mask image generated by the same script and overlay to achieve an iPhone-like glossy button.

iphone glossiness

Thanks!

Kind of long post, but I hope you're excited about removing a bunch of images from your future designs. If I've omitted some details, please let me know in the comments.

 

Give PNG a chance

Sunday, December 13th, 2009

Dec 13 This post is part of the 2009 performance advent calendar experiment. Stay tuned for the articles to come.

People are often afraid to use PNG because they think that:
a/ it doesn't work in all browsers, or
b/ filesizes are bigger than GIF

While these have some grain of truth to them, they are mostly misconceptions. Before addressing them, one quick background point - what's PNG8 and why it's cool.

PNG8

There are several types of PNG files, which can be grouped into those main kinds:

  • Truecolor PNGs with or without alpha transparency channel, also known as PNG24 and/or PNG32 (the one with alpha)
  • Grayscale PNGs with or without alpha
  • Indexed PNG, aka palette PNG, aka PNG8

PNG8 is like a GIF - it has a palette of 256 colors and supports transparency. While GIF supports true/false transparency (a pixel is either transparent or it isn't), PNG8 supports variable alpha transparency. Right there you see - PNG8 can do anything that GIF can, plus more.

There's a little glitch in IE6 where a semi-transparent pixels in PNG8 are seen as fully transparent, just like a GIF. So here's an option for progressive enhancement - you use the same image and IE6 gets a degraded GIF-like experience, while modern browsers get the full experience.

Here's an example, taken from this excellent article - modern browsers get the light bulb with the glow:

IE6 and under get the gracefully degraded experience and no glow:

Another pain point is that Photoshop doesn't produce semi-transparent PNG8 (although they came up with the name "png8" instead of saying palette or indexed PNG). Only Fireworks does export alpha transparent PNG8, which makes it a bit of a challenge. You also need a good designer to undertake this tricky task of making sure the image looks OK in both experiences. One way is to assume you're working with a GIF and then upgrade the experience with the carefully selected semi-transparent pixels. It could also help you keep the gif-like version in a layer and use other layers for the semi-transparent stuff, so you can quickly preview what the image will look like in IE6.

In any event - the important thing to remember is that in the worst case (IE6) PNG8 is as good as a GIF.

PNG doesn't work in browsers?

PNG works in browsers since forever with the exception of two edge cases:

  • the glitch where PNG8's semi-transparency is gone in IE6 (see above), but here GIF can't help you either
  • transparency in truecolor PNGs is shown as a solid (usually grey) color in IE6. But again - GIF can't help here either, because it doesn't support alpha (variable) transparency to begin with. People often use GIF to "solve" this problem (moving to GIF will mean potentially losing colors), but if you can solve with a GIF, you can solve it even better (and with smaller filesize) with PNG8

Another solution to the second problem is to use IE's AlphaImageLoader CSS filter (and there's a number of scripts to do so automatically), but this filter has serious performance drawbacks and should only be used as a last resort. Three things to try before resorting to AlphaImageLoader:

  1. Try PNG8 for progressively enhanced experience
  2. Try without transparency - if the background is a solid color, convert the image to use the solid color. In imagemagick you can use -flatten for this purpose:
    $ convert source.png -flatten -background yellow result.png
  3. Forget about IE6 :)

If you end up using AlphaImageLoader, make sure you use the underscore hack so that only IE6 users experience the performance degradation.

#some-element {
    background: url(image.png);
    _background: none;
    _filter:progid:DXImageTransform.Microsoft.AlphaImageLoader(src='image.png', sizingMethod='crop');
}

PNGs are bigger than GIFs?

This misconception comes from the fact that people compare truecolor PNG with GIF which is not a fair comparison because you often compare image with thousands of colors (the PNG24) with an image with 256 colors (GIF). Often people work on an image in Photoshop or another program and when they decide to export for the web, they try PNG24, see that it's bigger and switch to GIF. But in this step GIF may strip a lot of colors. And if you're going to strip colors, well, PNG8 will give you the same colors and smaller filesize. (Another thing is that sometimes Photoshop does a poor job exporting the PNG8. If the PNG8 looks crappy, but the GIF is OK, then export as a GIF but then convert to PNG with another utility, such as optipng)

Again - PNG8 is the file format comparable to GIF and it's almost always smaller in filesize than GIF.

Comparing GIF vs. PNG filesizes

(This and the next experiment is something I did over an year ago, bored to death in the middle of the ocean on the board of a Carnival cruise ship, but since then I never really looked at the data. So here's my chance to flush some old data and clean up 20 Gigs of lets-keep-just-in-case test images :) )

Using Yahoo! image search web service I downloaded some GIFs (matching the queries "logo", "mail" and "graph"), ended up a little over 1700 images. Then I used optipng to convert them all to PNG and see the results.

I used OptiPNG simply with no special options:

$ optipng *.gif

As the next experiment will show, optipng can do better, so can pngout for example. So consider these results the least you can do to make GIFs smaller (by turning them to PNG)

So some stats from the experiment:

  • The average, median actually, GIF image on the web (last year, judging from this small sample) is 525x388 and has 139 colors (I just love semi-useless stats ;) )
  • The median GIF is 24K
  • After conversion to PNG, the median becomes 18K
  • The median savings from converting all GIFs to PNG is about 23%

Interestingly enough 4% of the images were smaller as GIFs - utter disappointment (and don't tell anyone!). So I had to try just a little harder. I didn't run OptiPNG with its best -o7 option, but ran PNGOut instead. The results is that now only 4 of the 1706 images were smaller as GIF. I'm pretty sure that trying a little harder (with PNGSlim, see yesterday's post) would've probably fixed it, but 4 out of 1700 is something I could live with. BTW, the images where OptiPNG failed to produce smaller PNG, then PNGOut converted with the ratio of 21% median savings. Not bad for taming the few shrew GIFs.

BTW, some GIFs lost over 100K of filesize, the max was over almost 600K savings! So, you never know.

If you like to look at numbers, here's a csv dump - the optipng results and the selected few that ran through PNGout.

So, take-home message: turn your GIFs into PNGs and win at minimum 20% fewer bytes over the network.

Comparing PNG optimizers

For this experiment I downloaded over 12000 images (again, Yahoo! search API) and ran them through a bunch of optimizers, sometimes with different options. In retrospect, it's probably not that useful of an experiment, because (see previous post) different optimizers specialize in different areas - compression, pre-compression filtering, chunks removal, etc, and your best bet is to run several tools. But still it's at least some data points (the cvs dump is here)

The images were 1000 matches for each of the searches for "baby", "background", "bkg", "flower", "graph", "graphic", "icon", "illustration", "kittens" (of course), "logo", "monkeys", "png", "transparency". After removing 4xxs, 5xxs and other mishaps and cleaning up a bit, I ended with over 10000 images. I ran them thorough:

  • pngcrush - pngcrush -rem alla -reduce before.png after.png
  • pngcrush-none - to keep all chunks pngcrush -rem none -reduce before.png after.png
  • pngcrush-brute - more filter attempts - pngcrush -rem alla -brute -reduce before.png after.png
  • pngout - pngout /q /y /force before.png after.png. default compression level in PNGOut is "extreme", so I tried two less extreme below
  • pngout-match - pngout /s2 /q /y /force before.png after.png
  • pngout-intense - pngout /s1 /q /y /force before.png after.png
  • pngrewrite - pngrewrite before.png after.png PNGRewrite only works with PNG8, it also converts truecolor to PNG8 whenever the truecolor happens to be under 256 colors,
  • optipng - optipng before.png -force -out after.png. OptiPNG's default level is 2 (of 7) so I had to try below and above the default:
  • optipng1 - optipng before.png -o1 -force -out after.png
  • optipng3 - optipng before.png -o3 -force -out after.png
  • optipng7 - optipng before.png -o7 -force -out after.png
  • advpng - cp before.png after.png; advpng -z -f -q after.png
  • advpng-insane - with the "insane" 4th level of compression cp before.png after.png; advpng -z4 -f -q after.png
  • deflopt - cp before.png after.png; deflopt -s -f after.png
  • pngoptimizercl -cp before.png after.png; pngoptimizercl -file:"after.png"

And the results:

Tool Median time to run Median savings Success rate
pngcrush 0.25s 6.06% 93.85%
pngcrush-none 0.23s 5.58% 90.22%
pngcrush-brute 3.08s 8.10% 96.31%
pngout 1.89s 12.21% 94.35%
pngout-match 0.22s 13.89% 44.57%
pngout-intense 1.63s 12.10% 94.22%
pngrewrite 0.07s 29.84% 22.37%
optipng 0.23s 7.32% 93.21%
optipng1 0.10s 4.24% 85.16%
optipng3 0.66s 7.10% 94.26%
optipng7 4.13s 7.57% 94.81%
advpng 0.34s 11.55% 52.47%
advpng-insane 0.76s 15.64% 56.09%
deflopt 0.34s 0.44% 96.94%
pngoptimizercl 0.48s 9.71% 97.99%

"Success rate" is how often the tool managed to produce a smaller result than the original. For example PNGRewrite's success rate is pretty low, because it only works with up to 256 colors. Median time to run is the median value that the tool takes to optimize one image.

And now, madames et monsieurs, introducing...

Give PNG a chance (.com)

I hope you'll find this as funny as I do, I thought it was pretty funny, at least in my head :)

My secret goal was that everybody who hears the song or watches the video, will think twice the next time when doing "Save for the Web..." in Photoshop.

Enjoy!

Music: Drums from GarageBand, I play two guitars, also bass (a guitar with effect actually) and vocals. If you think you hear a woman's voice, it's still me, with "Helium" effect. The MP3 is here. If you want to experiment with the song yourself, here's a zip with each channel as an MP3.

Video: It may be lousy, but it's all web dev :) It's all JavaScript and CSS. The video is a screen capture of the Safari window. Also there are no images, only HTML entities. Heavy use of -webkit-* animations and transitions. The source and a live version you can play in Safari is here. The StarWars-like effect is borrowed from here.

The http://givepngachance.com URL is currently pretty blank, but I intend to add more PNG-related stuff there. Oh, and the lyrics.

Thanks!

Thanks for reading. And watching. And listening. Peace. And give PNG a chance :)

 

Big list of image optimization tools

Saturday, December 12th, 2009

Dec 12 This post is part of the 2009 performance advent calendar experiment (12 articles down, 12 more to go). Stay tuned for the articles to come.

Let's continue the topic of reducing file sizes started with the previous post and talk about making images smaller.

Engineer's guide to smaller images

Just to set the frame of the discussion - this is not about using Photoshop or setting the quality of the JPEGs and so on. I realize that we, web developers, wear many hats - we're designers, client/server coders, Apache/Linux admins, database heros. But this post is not about using image programs and assumes that you or your designer has already created the images to be used on the site with the appropriate colors, quality and so on.

Now, repeat after me: you should never take an image from the designer and put it up on the web.

Most often this image is bigger than it should be. It's not the designer's fault, it's usually the software used to produce the image.

You shouldn't put an image up on your server before running it through a few tools. These tools are free, open source, cross-platform and can be run on the command line, hence scripted and run in batches over a large number of files - by you, or even better, automatically by a build deployment process. It's OK to batch-run those files without human intervention, because these tools simply optimize the files, they don't change the pixel information, so the "after" images look exactly like "before", only smaller.

Selecting the right file format

The first step towards leaner images is to select the correct file format. There are three options:

  1. JPEG for photos. Photos contain millions of colors and smooth transitions of colors. Blue skies, clouds, sunset, your dog, lolcats - all photos.
  2. GIF is for the occasional "loading..." progress animation. This is it, no other uses for GIFs.
  3. PNG is for everything that's not a photo or an animation. That includes all icons, graphs, buttons, gradients, and what not. Any image with sharp transitions of colors. Think (but don't use for) text. Sharp transitions become "dirty" in JPEG.

PNG is an interesting topic for a follow up post, for now let's stop here. If it's not a photo, it should be PNG. An edge case is a screenshot for example. Depends on what's on the screen of course, but most often JPEG will give a smaller size if you can live with the artifacts around the sharp edges (like text).

Optimizing GIFs

Unfortunately many people still use GIFs even for non-animated images. That's a mistake. PNG is a superior format and yields smaller file sizes.

People still use GIFs because they think either that a/ GIF is smaller than PNG or b/ there's lack of support for PNG in browsers. These are misconceptions and I'll talk more about them tomorrow.

So, the way to optimize a GIF is to convert it to PNG.

You can use many tools to turn your GIFs to PNGs, including ImageMagick and OptiPNG.

# option 1: ImageMagick (if you know the filename)
$ convert logo.gif logo.png

# option 2: ImageMagick again (if you just convert all files in a directory)
$ mogrify -format png *.gif

# option 3: OptiPNG
$ optipng *.gif

These are just some of the options, I'm sure there are others.

After the conversion you can optimize the new PNGs like all other PNGs

Optimizing PNGs

There are various ways to write a PNG file. Unfortunately not all image editing programs do a good job at writing PNGs for the minimal file size.

Luckily, to fill the void, there's a great number of tools that excel in writing small PNGs. There are different ways to optimize a PNG:

  1. Stripping out "chunks" - PNG is an extensible format. Extensions come in the form of chunks and most chunks are not needed for the web.
  2. Reducing the number of colors and switching between PNG types - truecolor PNG, grayscale, palette...
  3. Chosing the best "filter". Filters are a pre-compression step. You can compress any type of file, but when you know the file is an image, you can do better. Filters are for this purpose.
  4. Optimizing the actual DEFLATE compression algorithm

Different tools specialize in one or more of these areas. So the more tools you run, the better the results will be. But you have to run at least one tool, always. You'll be surprised how unoptimized are most PNGs coming from common commercial image programs.

So, to optimize a PNG you shoul run as many of the following programs as possible:

# optipng (skip -o7 to run faster)
$ optipng -o7 my.png

# pngcrush (skip -brute to run faster)
$ pngcrush -rem alla -brute -reduce my.png my.png.temp
$ mv my.png.temp my.png

# pngout - closed source, non-windows binaries here
# (add parameter -s2 to run faster)
$ pngout my.png

# advpng (use -z2 to run faster)
$ advpng -z4 my.png

# deflopt - windows only
$ deflopt my.png

Other tools to note include PNGrewrite, PNGNQ and PNGquant, but they are limited because they deal only with PNG8 (256 colors) files. PNGNQ and PNGQuant are actually converters from truecolor to PNG8, so they are not guaranteed to be lossless. PNGreqwrute is safe to use, it will just silently fail if the file has more than 256 colors, so there's nothing to lose.

Oh, and another, excellent tool - PNGOptimizer, windows-only has both command line interface and a GUI.

PNGSlim for the hardcore PNG optimization

If you're really serious about optimizing your PNGs, the tool is called PNGSlim. It's a Windows-only batch file that runs pretty much all tools above and runs them (especially PNGOut) with all kinds of parameters, hundreds of times. So it can take a while to run.

Optimizing JPEGs

JPEG is a lossy format (you lose information every time you save it, even if you choose 100% quality), but there are some operations that can be done losslessly - such as tweaking comments and meta information, cropping, rotating to 90, 180, 270 degrees. The tool that does this magic is called JPEGTran and is likely already on your unix/linux box. If not - here's how to install it (for Windows - get the .exe here)

So to optimize a JPEG losslessly you remove the meta information and optimize the so called Huffman tables. For bigger JPEGs (bigger than 10K) you can also convert the image to progressive coding.

# strip meta and optimize
$ jpegtran -copy none source.jpg > destination.jpg

# strip meta and convert to progressive coding
$ jpegtran -copy none -progressive source.jpg > destination.jpg

# keep all meta but still optimize
$ jpegtran -copy all source.jpg > destination.jpg

Important note on stripping meta

Only strip meta information from images you own the rights for and have permission. Otherwise you're committing a crime. Photographers put important copyright information in meta markers.

Optimizing GIF animations

Remember - no GIFs other than animations. For animations, run GIFsicle (pronounced "yo' mama" :) ) berfore you put them up:

# GIFSicle
$ gifsicle -O2 source.gif > destination.gif

More tools?

These are the core tools for image optimization. There's a number of wrapper tools that are more or less UIs on top of these, because there are people who don't like consoles (really?!). I'll list the few that I can think of, please comment if you know of others, especially for windows. It's nice to give nice UIs to give to designers so they can drag-drop optimize images too.

  • smush.it, created by yours truly and Nicole Sullivan, now part of YSlow - runs pngcrush, jpegtran, gifsicle
  • PageSpeed runs optipng, jpegtran
  • PunyPNG - originally inspired by smush.it, but more advanced
  • ImageOptim - Nice easy UI for Mac, runs most of the tools above (hope your company firewall doesn't block the site because of the domain name ;))
  • PNGSquash another UI for Mac, runs advpng, pngcrush, optipng
  • PNG Monster is for Windows, runs many PNG tools, you can drag/drop on it
  • IrfanView - my favorite image viewer for Windows has a plugin to use PNGOut
  • WP-Smushit is a WordPress plugin by Alex Dunae which sends all your image uploads to smush.it for optimization. Talk about easy!

More reading

Series of articles on YUIBlog:

presentations:

and more:

Thanks!

Thank you for reading. Now you have a whole lot of tools/toys to install and play with. Image optimization is an easy way to improve performance, it's just running a bunch of tools. You don't need to worry that the quality will suffer (so the designer won't be disappointed in you :)). So you can only win. You may win quite a bit, you may win just a little (anywhere between 5 to 30% savings is what I've seen on random live sites when I was working on and testing smush.it). The thing is you'll almost always win something.

And because it's human to forget to optimize the images before you push them live, do take the time to setup the optimization step as part of the automated deployment process.

And to summarize once again the steps of what this automated process would be:

  1. Convert GIFs to PNG. Then relax and take a deep breath (instead of flaming the unfortunate soul who created the GIFs) while you string replace "gif" with "png" in all your styleshets
  2. Run PNGs through optipng, pngcrush, pngout, any or all of the tools listed above
  3. Run JPEGtran
  4. Run GIFsicle
 

Duplicates and near-duplicates

Wednesday, December 9th, 2009

Dec 9 This post is part of the 2009 performance advent calendar experiment. Stay tuned for the next articles.

One of Yahoo!'s first batch of performance best practices has always been "Avoid duplicate scripts" (check Steve Souders' post). Later we added "... and styles". This is a pretty obvious, kind of a "Duh!" type of recommendation, it's like saying "Avoid sleep() in your server-side scripts". But it didn't come up out of thin air, duplicates were noticed on some quite high-profile sites.

Duplicates are easy to spot (and YSlow will warn you), but let's talk a bit about another concept - let's call it near-duplicates - when two components are similar, almost the same, but not quite.

Duplicate scripts and styles

As a refresher and a quick illustration of the effects of duplicate scripts and styles, fire off your HTTP sniffer and hit this test page.

(btw, this is a simple page I put up to test different YSlow scenarios, you can actually use it as a web service of sorts to create any type of components with different options)

Firefox 2 downloading both duplicate styles and scripts:

FF2 duplicate scripts and styles

IE6 and duplicate scripts:

IE duplicate scripts

Exact details of when/which browsers chose to download duplicates are not that interesting, it's obviously bad to waste time downloading the same resource. Even if no repeating download happens, the browser still has to parse through and execute the script/style for a second time.

Even if you have iframes you don't need to repeat the same JS/CSS in each frame, you can "borrow" them from the parent page, here's an example.

Near-duplicates

Near duplicates can be:

  • components with the exact same response bodies but different URLs causing the browser to do double work
  • components (images) that are too close to each other - in terms of looks or purpose. Only one component should be selected in this case.

Same component, different URLs

This could happen especially when you have user-generated content such as image uploads for profile photos and avatars in social sites, forums, images people put in comments on MySpace and so on.

Also images of stuff for sale (Craigslist, eBay). Often different sellers offering the same item would take the same photo from the manufacturer's site and upload it over and over again.

Luckily, PageSpeed warns about components with identical content, so those can be identified:

In the screenshot above, you see one image (2.3K) repeated 3 times, another (the iPhone, 1.7K) is repeated 4 times, and yet another one (2.8K) repeated 2 times.

It's not exactly trivial to avoid this type of duplications with user-generated content (for example, the first poster may delete the photo, in which case the second poster's photo will need to "shine through"). But it's not impossible, using for example a hash of the component's content as an identifier.

Loading...

Ajax loading indicators are a great idea to give feedback to the user that something is happening. They come in all shapes and sizes... sometimes on the same page, unfortunately. And again, sometimes it's the same stock image but used at different stages of gradual "ajaxification" of the page and with different URLs.

As we're moving more and more towards modular pages and client-side logic, often different modules on the same page are coded by different teams at different times, independently, without being aware of each other's assets. This way of building pages has it's challenges and one is that common components, such as Ajax loading indicators, should be shared.

Too similar modules

Along the same lines - different modules are sometimes created by different designers at different times. The result - one rounded corner box with 1px shadow and one with 2px shadow, both on the same page. Or two different shades of the same gray color, which no one can tell apart. That's just a waste. (See Nicole Sullivan's presentation for illustration, e.g. slides 44, 45)

Below is an example, can you tell that these 5 rounded corner boxes are all different - slightly different shadow, color or radius? How many different boxes does this page need?

Different sizes of the same image

It's highly recommended to not scale images in HTML (or CSS). If you need a 100x100 image you don't use a 400x400 one with <img width="100" height="100" ... />. That's a good rule of thumb... to break sometimes ;)

In cases where the same image is used with different sizes and likely even on the same page, it may be beneficial to reuse the same bigger image and scale it down, because this could be saving extra HTTP requests of downloading the same (but slightly smaller) image.

Facebook is an example, the same hairy guy on the screenshot has two images with different sizes. It's actually the same image but resized in CSS.

The relevant CSS which shows the profile image in LARGE and SMALL (and looks like there's even a TINY view, although I couldn't find an example on this page)

.UIProfileImage_LARGE{width:50px;height:50px}
.UIProfileImage_SMALL{width:32px;height:32px}
.UIProfileImage_TINY{width:25px;height:25px}

Thank you!

Thanks for reading! Reducing HTTP requests is critical for page performance. You've merged your scripts and styles as much as reasonable, you've crafted CSS sprites and inlined images with data URIs. Now it's time to look at what's left - are there components that are way too similar, are there any near-duplicates or even exact duplicates? Same image on different backgrounds? Ever-so-subtle gradients and shadows? Time to pick up the old axe and cut.

 

Data URIs, MHTML and IE7/Win7/Vista blues

Monday, December 7th, 2009

Dec 7 This is the seventh in the series of performance articles as part of my 2009 performance advent calendar experiment. Stay tuned for the next articles.

Let's start this post as a dialog:

- Data URIs are a way to embed the base64-encoded content of images inside HTML and CSS. Inlining images in markup or stylesheets helps you save precious HTTP requests. It's an alternative to CSS sprites.
- BUT! IE6 and IE7 don't support data URIs
- Yes, for them you can inline the images using MHTML
- BUT! Looks like IE7 on Vista and IE7 on Win7 have problems with MHTML
- [Sigh...] For those browser/OS combos there's a solution too.

I'll quickly go over what are data URIs and how MHTML addresses IE6 and IE7. If you're familiar with these, feel free to skip down to the Vista blues part.

Data URIs

Here's how to use data URIs in your pages:

  1. take an image:
    horoscopes icon image
  2. Use PHP to read this image's binary content and encode it using base64 encoding:
    $ php -r "echo base64_encode(file_get_contents('horoscopes.png'));"
    iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAMAAADXqc3KAAADAFBMVEX///8mPnru9...
    ...more scary stuff goes here...
    dajNGGzlDAAAAABJRU5ErkJggg==
  3. To use this image in CSS, paste the encoded image content in the stylesheet following this syntax:
    .horoscopes {
        background-image: url("data:image/png;base64,iVBOR...rkJggg==");
    }

    Note: the trailing == is part of the image content it's not part of the syntax

  4. Alternatively if you want to use this image in the HTML, you can go like:
    <img src="data:image/png;base64,iVBOR...rkJggg==" />

And that's about it for data URIs, it's quite simple actually.

Data URIs in the wild

To see real-life, high-traffic sites using data URIs look no further than your favorite search engines.

Yahoo! Search using data URI in CSS for a button background:

yahoo search screenshot

Google Search using data URI in an IMG tag for a video thumb:

Google search screenshot

Base64-encoded filesizes

The base64 encoding adds about 33% to the filesize. But, then you gzip the result, the compression brings the size back to the original. Plus or minus. Interestingly enough, sometimes after base64 and gzip, the result is smaller than the original. But don't count on that.

Theoretically also when you base64 encode a bunch of images and inline them in the same CSS or HTML, you'll have a larger string to compress, so increasing the chance of repetitions and compressing better. (Todo: test this assumption)

In any event, the benefit of reducing HTTP requests will likely greatly outweigh any fluctuation in the file size.

MHTML

IE8 supports data URIs, but earlier IEs do not. For them you can use MHTML (Multipart HTML, blogged previously here)

Continuing with the previous example, in order to display the horoscopes image in IE < 8 you can use a stylesheet like this:

/*
Content-Type: multipart/related; boundary="_MY_BOUNDARY_SEPARATOR"

--_MY_BOUNDARY_SEPARATOR
Content-Location:horoscopes
Content-Transfer-Encoding:base64

iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAMAAADXqc3KAAADAFBMVEX///8mPnru9....U5ErkJggg==
*/

.horoscopes{*background-image:url(mhtml:http://example.org/styles.css!horoscopes);}

A few notes on this example:

  • the image content is base64-encoded just like before
  • if you need more images, use your chosen separator prepended with --, in this case --_MY_BOUNDARY_SEPARATOR
  • Content-Location:horoscopes is how you give a name to the image in order to use it later
  • then later in your stylesheet you can reference the name in the stylesheet using http://url/styles.css!horoscopes
  • you need absolute URLs in the mhtml:http://.... part (that's retarded, hope I'm mistaken. Didn't work for me with relative URLs)
  • you can use a single CSS file to support both old IE as well as new browsers, but you'll have to repeat the image stream twice, probably a better approach is to use browser specific stylesheets

For more examples of MHTML (which is also used for multipart email messages) check this and for a brilliant example of using one MHTML to embed images in HTML (not CSS), check Hedger's test page here (appears 404 at the moment of writing). For a tool to automate the dataURI-zation/MHTML-ization, be sure to check Nicholas Zakas' CSSEmbed

The problem with IE7 on Vista (and Win7)

So far we have data URIs and MHTML fallback. Turns out we're not done yet, because IE7 on Vista and Windows 7 fails with the MHTML example. Two points:

  • IE8 is the default in Windows 7, but it comes with several IE7 modes (either by default or turned on by users when their favorite pages break). Compatibility view, ie7 mode, ie8 in ie7 mode, blah-blah, it's all pretty confusing, but all modes with the exception of IE8 in proper IE8 standards don't work with the MHTML
  • I tested Windows 7 evaluation copy, but I have the bad, bad feeling that Windows 7 normal copy will be as broken when it comes to MHTML. I believe it has something to do with security, if anyone finds an article on MSDN, or anywhere, please share.

The solution to the Vista problem is to split the CSS shown above (the one that uses MHTML) in two: one which contains the encoded images (the commented part) and a second one which only has the normal CSS selectors part.

In other words have something like:

  1. mhtml.txt:
    Content-Type: multipart/related; boundary="_MY_BOUNDARY_SEPARATOR"
    
    --_MY_BOUNDARY_SEPARATOR
    Content-Location:horoscopes
    Content-Transfer-Encoding:base64
    
    iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAMAAADXqc3KAAADAFBMVEX///8mPnru9....U5ErkJggg==
    

    No need to use comments. Could be .txt, .css or anything you like. In my tests the content-type didn't matter - text/html, text/css, text/plain all worked

  2. styles.css:
    .horoscopes{*background-image:url(mhtml:http://example.org/mhtml.txt!horoscopes);}

    The normal CSS simply references the new MHTML file appending !identifier

  3. in your HTML you don't need to reference the MHTML document, you just point your link tag to styles.css as usual and styles.css contains the reference to the MHTML doc

Time to celebrate? Almost.

Another ugly bug surfaces - this procedure outlined above works only once. Once the MHTML.txt is cached, it stops working. That means repeating visits to the page or other pages using the same mhtml. The effect also means hovering on the same page. Say you have two images - one normal and one mouse over, both in the same MHTML. The normal works and the hover breaks, mouseout breaks too. Insane, isn't it?

There's a solution though - you should make the browser request the MHTML document every time. This is kind of backwards when it comes to performance optimization, where we want to cache everything. It's pretty unfortunate. The best you can do is avoid sending the MHTML every time, but only send a "not-modified" header. In order for this to work, the browser has to send If-Modified-Since or If-None-Match.

So you can send your MHTML file with an ETag (yes, ETags can help sometimes ;)) Then the browser will send an If-None-Match and you can happily reply "304 Not Modified".

Sounds complicated? It sure is. And, remember, all this is only for IE7 (or any IE7 mode) on Vista and Windows 7. All other IEs on all other platforms work with the easy MHTML and IE8 works with data URIs.

Everything is a trade-off (see last night's post) and I can understand how you may think "That is one big tradeoff". There's always the option of serving normal images to the affected browser/os and just don't implement this optimization for them.

But it's good to know there is a solution.

DataSprites class

Let me offer this DataSprites PHP class I coded that takes care of all the scenarios. It takes a bunch of image filenames as input and produces:

  • a CSS containing data URIs for normal browsers (example output)
  • an MTHML CSS fallback for IE6,7 (example output)
  • a CSS that refers to an additional MHTML for IE7 on Vista, Win7 (example output - css and mhtml). The MHTML in this case is sent out with an ETag (derived from the input image filenames) and consecutive requests with the same ETag return 304 Not Modified

You can see a test page here and view the source code here. The directory listing is also available if you want to grab the images for testing.

The perfect use case for such an approach is when you want to combine background images on the fly. When you would normally use CSS sprites, but you want to produce them dynamically at run time (maybe because you have too many combinations?).

I've tested this in Safari, Firefox, Opera, IE6, IE7 and IE8 (in all compat modes) on Vista and Windows 7 evaluation copy. If you find the test page is not working for you, please let me know.

In this DataSprites class, I'm checking the problem OS looking for the strings "Windows NT 6" and "Windows NT 7" in the user agent string. My Win7 evaluation copy had "Windows NT 6.1" in the UA, so I may be a little forwards-aggressive with the check, but somehow I thought Win7 proper should have "Windows NT 7" in the UA string. And also that Win7 will suffer from the same issue as Vista and Windows 7 evaluation.

UPDATE: Added a hover example.

UPDATE 2: Recorded a screencast to demo the IE7/Vista experience

Thanks!

Thanks for reading! Now you know all about data URIs, MHTML, and maybe a little too much about IE7/Vista's challenges :) Ready to use data URIs and save some HTTP requests?

 

Reducing the number of page components

Saturday, December 5th, 2009

Dec 5 This is the fifth in the series of performance articles as part of my 2009 performance advent calendar experiment. Stay tuned for the next articles.

Let's talk a but about waterfall optimization - the first thing that happens in Mr.Page's life. The best way to optimize and speed up the waterfall is to have less stuff in it. The fewer page components, the faster the page - simple as that.

Fewer components vs. component weight

The size of the page components, meaning their size in kB, is important. It makes sense - smaller pages will load faster, 100K JavaScript will load faster than 150K. It's important to keep sizes low, but it should be clear that the number of components is even more important than their filesize.

Why? Because every HTTP request has overhead.

OK, but how bad can it be, someone might ask. If you look at an HTTP request - it has a header and a body. A 100K body will greatly outweigh the size of the headers, no matter how bloated they are.

Here's the headers for a request to Yahoo! Search:

Host: search.yahoo.com
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5;) Firefox/3.5.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

To that request the server responds with the body (content) of the response prepended with some headers like:

HTTP/1.1 200 OK
Date: Sat, 05 Dec 2009 07:36:25 GMT
P3P: policyref="http://p3p.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR... blah, blah"
Set-Cookie: sSN=nTMt3Lo2...crazy stuff...nLvwVxUU; path=/;domain=.search.yahoo.com
Cache-Control: private
Connection: close
Content-Type: text/html; charset=ISO-8859-1

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">

<html lang="en"><head><meta...

This is 352 bytes request header and 495 bytes response header. Not so bad, eh? No matter how hard you try to make your cookies monster-size, the response body (9k gzipped in this case) will always be significantly bigger. So what's the problem with the overhead of the HTTP requests then?

The size of the headers is a problem when you make requests for small components - say requests for small icons - for example 1K or under. In this case you exchange 1K headers to get 1K of useful data to present to the user. Clearly a waste. Additionally this 1K of headers can grow once you start writing more cookies. It may very well happen that the HTTP headers size is bigger than the actual icon you need. And even if the headers are not bigger that the component, they are still big when you think percent-wise. 1K of 10K is 10%.

But the HTTP headers size is only one (and the smaller) of the problems.

The bigger problem is the HTTP connection overhead.

HTTP connection overhead

What happens (at a high level) when you type a URL and hit Enter? The browser sends a request to the server. Which server? The browser needs to know the IP address of the server, so if it doesn't have it in the cache, it makes a DNS lookup. Than the browser establishes a connection to the server. Then it waits for the first byte of the response from the server. Then it receives the full response (payload).

Here's how this looks like graphically represented by webpagetest.org

And the color legend:

  1. DNS lookup
  2. Initial connection
  3. TTFB (Time to first byte)
  4. Payload

So what do we have here - looks like in this particular case the browser is downloading content about 40% of the time. The rest of the time it's... well, not downloading content. How's that for an overhead. And the part of not downloading can be even greater, this above was just one example.

Now how about this - a bird-eye view of the 4 last pages in webpagetest.org's test history - just some random pages people have tested.

Do you see a lot of blue (time spent downloading content). Not as much as you would've hoped. There's some DNS lookups, some orange... and OMG, talk about going green! :)

In fact you may notice that the smaller the component, the smaller is the blue part.

What does all this tell us?

  1. A significant chunk of time is spent in activities other than downloading.
  2. Smaller components still incur HTTP overhead and for them the relative penalty (relative to their size) is atrocious.

So what's a performance optimizer to do? Reduce the number of components and so pay fewer penalties.

Just remove stuff

Truth is - a lot of stuff on the pages today is not needed. Features no one likes or uses clutter the page and make it heavier. Well, what can you do, the boss/client/marketing guy wants that feature there. What you can do is at least try. You can introduce some science into the marketing activities - measure how much a specific feature is used. Or if you already have the data - look at it. Decide what can a page go without.

It's going to be tough convincing people to remove stuff. After all you spend time developing it. Someone dreamt up that feature initially. Someone (not the users) loves it. People hate to let go. But still, it's worth to try.

Combine components

Now that the phase of convincing people to remove stuff is done, what's left needs to be combined. How do you combine components? Simple - all JavaScripts go into a single file, all CSS into a single file. All decoration images go into a sprite.

JavaScript example (from a page to remain anonymous)

Before:

<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.2.6/jquery.min.js"></script>
<script src="/javascripts/application.js?1258423604"></script>
<script src="/javascripts/ui/minified/jquery.ui.all.min.js?1258423604"></script>
<script src="/javascripts/ui-ext/ui.bgiframe.min.js?1258423604"></script>
<script src="/javascripts/ui-ext/ui.stars.pack.js?1258423604"></script>
<script src="/javascripts/ui-ext/ui.dimensions.js?1258423604"></script>
<script src="/javascripts/ext/jquery.form.min.js?1258423604"></script>

After:

<script src="/javascripts/all.js"></script>

Sizes, gzipped: 70029 bytes before, 65194 bytes after. Simply merging files and there's even 6.9% saving!
And the more important saving: 6 less HTTP requests

Repeat for CSS. Before:

/stylesheets/general.css?1258423604
/stylesheets/global.css
/stylesheets/ui.stars.css
/stylesheets/themes/enation/enation.all.css
/public/template/css/132/1245869225

After:

<link type="text/css" rel="stylesheet" href="/stylesheets/all.css" />

Sizes, gzipped: before 14781 bytes, after 13352 bytes, saving 9.6%.
But the bigger saving: 4 less HTTP requests.

If you wonder how come the sizes before and after are different since we merely concatenate the contents of the files, well, the savings come from gzip compression. When you have more characters in the file, there's more chance that some will repeat which means they will compress better. That's one. And then, the compression itself has an overhead which you incur once for the whole bundle of files, as opposed to for every file.

Now - let's make decoration images into sprites. Before:

... 15 image requests, 6.8K

After: (1 sprited image)

Result size: 1.4K, 7 times smaller!

Here the savings are so dramatic partially because the source files are GIFs and the result is a PNG8 but that's a whole other post.

So in conclusion: file concatenation is just awesome. You save both: bytes to download and, much more importantly, HTTP requests. Less of the green stuff in the waterfall!

x-type component concatenation

So far we combined .js with .js, css with css and images with images. How about a cross-component type concatenation?

You can inline images inside HTML and CSS (and why not JS if you want) using data URIs (another post coming).

And you can inline CSS and JS inside HTML too.

This means you can have your whole application inside of just one HTML file if you wish. Inside of the HTML you have inline styles, scripts and images.

Combining CSS with JS

Now how about mixing CSS and JS into one component. You can do that, and it's especially suitable for lazy-loaded, widget-type functionality.

Say you've loaded the page, then the user clicks some rarely used button. You haven't downloaded that content that is supposed to wow the user upon click of the button. So you send a request to grab it. The new content may come in the form of a JSON string. And what if the new content requires some stylesheet that was not part of the base page? You'll have to make another request to download that stylesheet too.

Or, you can download both content and styles in the same JSON response. You simply inline the style information as a string in the JSON. So:

1. uses clicks, you request feature.js which goes like:

{"content":"<p class=\"wow\">I'm a feature</p>", "style": "wow{font-size: 60px}"}

2. You process the JSON and shove the content into the page

var o = JSON.parse(xhr.responseText);
$('result').innerHTML = o.content;

3. You add the styles to the head:

var wow = document.createElement('style');
wow.type = "text/css";
if (wow.textContent) { // FF, Safari
    wow.textContent = o.style;
} else {
    wow.styleSheet.cssText = o.style; // FF, IE
}
document.documentElement.firstChild.appendChild(wow);

Nice and simple. Makes the features (that progressively enhance the page) atomic and self-contained.

More to reducing components?

For more and creative ways to reduce HTTP components you can take a look at MXHR and Comet

Another thing to check is the Keep-Alive setting on your server. Remember how there were 4 steps in the component download. When you request a second component you can have the connection open so that you don't need to re-establish it (skipping step 2). And since the DNS lookup was already made you get rid of step 1. Skipping 2 out of 4 is not bad at all.

Summary

Reducing the number of page of components is the top priority of any web performance optimization effort. HTTP requests are costly. In part because of the headers size overhead, but mostly because of the connection overhead. The browser spends a disturbing amount of time not downloading stuff, and we can't allow this!

 

Performance Advent Calendar 2009

Monday, November 30th, 2009

I like 24ways.org's idea of a webdev advent calendar - publishing one article per day from Dec 1st to Dec 24. I thought it would be cool to have the same thing on web performance topics. Our young performance community would benefit from a year-end heartwarming bunch of articles, maybe some motivational, some wrapping up interesting ideas and techniques that popped up in the past year, this kind of stuff. Ideally it would be a co-effort by performance folks from around the world, but lacking the PM skills and time to organize it (and coming up with the idea kinda late) I'll try to start alone, hopefully next year it would be a community effort.

So this is now just a placeholder post, I'll update it with links to the articles as I write them, one per day is the plan, will see how it will work. So watch this space, you can follow me via the RSS feed or Twitter.

I have a preliminary idea of the articles I intend to write, but it's not at all final or complete, so if you want to see an article on something, or you want to write a guest post, please comment or get in touch in email.

  1. The performance roadmap
  2. Performance tools
  3. Required reading
  4. Psychology of performance
  5. Reducing the number of page components
  6. The pain points of having fewer components
  7. Data URIs, MHTML and IE7/Win7/Vista blues
  8. Collecting web data with a faster, free server by Christian Heilmann
  9. Duplicates and near-duplicates
  10. Caching vs. inlining
  11. Reducing the payload: compression, minification, 204s
  12. Big list of image optimization tools
  13. Give PNG a chance (video)
  14. Free-falling waterfalls
  15. JavaScript loading strategies by Ara Pehlivanian
  16. How To Measure Web Site Performance by Eric Goldsmith
  17. Rendering: repaint, reflow/relayout, restyle
  18. DOM access optimization
  19. The new game show: "Will it reflow?"
  20. Extreme JavaScript optimization by Ara Pehlivanian
  21. Progressive rendering via multiple flushes
  22. iPhone caching
  23. CSS performance: UI with fewer images
  24. The performance business pitch
 

Help write the lyrics to “Give PNG a chance”

Thursday, September 3rd, 2009

As you know yours truly is a guitar hero wannabe. So I'm playing with the idea of recording a song/video called "Give PiNG a Chance", cover of "Give peace a chance". Hopefully whoever hears it will then think twice before saving a GIF instead of PNG. Imagine. And the web will be as one... :D

Here's the lyrics so far, it's a work-in-progress. If anyone can think of rhyming Web 2.0 buzzwords, please comment.

Give PNG a Chance

#1
Everybody's talking about

Web 2
YouTube
WordPress
CMS
    Highrise
    Enterprise
    Crowd sourcing
    Open sourcing
Phone texting
Unit testing
Foo bars
Avatars

All we are saying is: give PNG a chance

#2
Everybody's talking 'bout

Long Tail
Junk Mail
Fail Whale
Ruby Rail
    Webcasting
    Screencasting
    Podcasting
    Vodcasting
Cloud computing
Tele commuting
Trackbacks
Pingbacks
    Less is More
    Apple store
    Link whore
    Browser war

All we are saying is: give PNG a chance

#3
Everybody's talking about
RSS, CSS
HTML, XML
    SOA, RIA
    API, CGI
REST-POST, GET-HEAD
SOAP? Nope!
    SEO, SMO
    CPM, CRM
SQL, YQL
SMS, MMS

All we are saying is: give PNG a chance
All we are saying is: give PNG a chance
All we are saying is: give PNG a chance
All we are saying is: give PNG a chance
Yup, no more GIFs
All we are saying is: give PNG a chance
All we are saying is: give PNG a chance...

Unused buzzwords

Here's my laundry list of buzzwords I thought of but didn't use so far in the lyrics. If you think of good rhymes as you look through the list, please comment.

  • flickr
  • blog
  • blogging
  • wiki
  • twitter
  • facebook
  • outlook
  • ajax
  • comet
  • firefox
  • chrome
  • ie6.. ie8
  • explorer
  • navigator
  • digg
  • blogger
  • safari
  • opera
  • RoR
  • PHP
  • CDN
  • SMS, MMS
  • WTF
  • FTW
  • web service
  • Perl, cURL
  • Ruby
  • Python
  • social
  • social media
  • social tagging
  • click-through
  • agile
  • photoshop
  • rounded corners
  • fluid layout
  • iPhone
  • Air, PEAR
  • Sliverlight,
  • mobile
  • JavaScript
  • flash
  • buzzwords
  • keywords
  • microblogging
  • microformats
  • micropayment
  • semantic web
  • validators
  • web standards
  • folksonomy
  • taxonomy
  • tagging
  • phishing scam
  • spam
  • widget
  • gadget
  • geotagging
  • linkbait
  • bookmarking
  • viral
  • tag clouds
  • view source
  • torrents
  • bit torrent
  • github
  • blog
  • mashup
  • pagerank
  • social network
 

Ignite Velocity: Image Weight Loss Clinic

Wednesday, September 2nd, 2009

A few months ago I gave a 5 minute Ignite talk at the Velocity conference. (My previous post talks about the Ignite experience.) I thought they recorded the video and wanted to share but seems like it's not happening. So below are the slides from slideshare.

The ignite talk rules are: 20 slides that change automatically after 15 seconds. But I took the liberty of hacking the format a little bit. For one, the 18th (and 19th) slide which was my main point is duplicated. And then I had some transitions, like in the first two slides for example, where different text or pictures change after a timeout. I expanded those transitions for slideshare, so the slide count is 40-something, but at the presentation it was still total of 20 slides in 5 minutes.

If you follow the slides on slideshare, be sure to click the Notes tab which has what I said, more or less, at each slide. The full script is below.

And yes, if you like it, please vote for my South by Southwest panel proposal. It has the same name "Image Weight Loss Clinic" but will be a much longer version of this 5 minute Iginite and what is more - Derek Tonn of graphicsoptimization.com will share the stage with me. Pretty exciting, so cast you votes, leave a comment and help me get to SxSW.

The Script

Here's what I said, more or less:

  1. Let’s talk a little bit about cutting back on that bandwidth bill. Saving some money. And without touching the code. How do we do that? One way is…
  2. Images. Ever since the dawn of times people have been using images. Cavemen scraping on their cave walls… and today… Today we have websites. And LOLcats.
  3. Half of the average web page weight today is images. And most of these images are too fat. They could use a diet and lose some weight but without losing quality, not even a single pixel.
  4. So, welcome to the Image Weight Loss Clinic. Our first patient today is... some random page found on the Internets. Hmm, what have we got here… Header graphic, speaker photos, logos, rotating banners – the usual stuff.
  5. First problem - there are some GIFs. We shouldn’t be using GIFs. PNG is the format for graphics - buttons, icons, and so on. The palette type of PNG, also known as PNG8 is good as a GIF and actually better. And PNG filesizes are smaller. Well, except for really tiny GIFs, but let’s ignore these, they don’t matter much.
  6. You should convert all your GIFs to PNG8. There’s a number of tools to do this in a batch. You can use ImageMagick for example. “Instant results or your money back”. Especially since ImageMagick is free.
  7. For your JPEG photos, use JPEGTran. It’s free, cross-platform, command-line tool, easy to script. And it’s lossless, you keep the photo quality. If you run it without setting any options, that’s good enough. Run with the –optimize and you’ll get even better results.
  8. Option –progressive makes JPEGs that render in the browser from low resolution to high, as opposed to top-to-bottom. If the JPEG is bigger than 10K (which is most JPEGs), the progressive encoding has a good shot at giving you a smaller image. You can always brute force – run with both options and take the smaller result.
  9. Option –copy none strips all comments and meta data - camera model, geo location, everything. If you own the image, strip the meta data. You’ll be surprised how much bloat is in there. If you don’t own the image, check with the owner. Option –copy all keeps all meta data.
  10. Next – PNG. There’s a lot of options to optimize PNGs - all lossless and scriptable on the command line. Some even have nice graphical UIs built on top of them, you know, for our command-line-challenged friends.
  11. There’s even browser-based UIs, built on top of some of the tools. For example, smush.it, which is now part of YSlow runs PNGcrush. Google’s Page Speed runs OptiPNG.
  12. With so many PNG tools, how do you decide which one to pick? They are all good tools and they do their best, so if you’re in a hurry, just pick one and go with it, script it. You’ll be pleasantly surprised by the outcome.
  13. Now if you’re serious about PNG weight loss, you can run all the tools you can lay your hands on. Run the next tool on the result of the previous, try with their different options switches. They all do different things, and results may vary from one image to the next. Run them all in sequence and get an even smaller file at the end.
  14. Or, if you’re hardcore – run PNGSlim. PNGSlim is actually a Windows batch file that runs most of the other tools. It’s very slow, it can take hours to go through a handful of files. But at the end, it will give you the leanest, most optimized PiNG.
  15. Let’s see some results. That page has a total page weight of over 800K, including scripts, styles, everything. Over 80% of the weight is images. And what happens after optimization?
  16. We can strip quite a bit, over 200K of unnecessary image information. 200K, that’s over 30% if the image weight. The page looks exactly the same, pixel by pixel. There was no manual work involved, just running some tools. I think 30% is pretty impressive.
  17. To summarize, don’t use GIFs, run JPEGtran on your photos, and for the graphics - any PNG or all PNG tools.
  18. So the tools are available out there, now all you need is a process that runs the tools before your images go live. Make image optimization part of the push process.
  19. These are all lossless operations that don’t require you to look at the images to verify they’re OK. So you can only win! And often you’ll see some amazing and surprising results. Even if you normally do a good job with image sizes, there’s always a few that slip as you race against the deadline. The thing is – setup a process, so you don’t even need to think about it.
  20. Thanks very much, everybody! I’m Stoyan from Yahoo! Search. Enjoy Velocity, it's a great conference!
 

The art and craft of postload preloads

Thursday, August 27th, 2009

What can your tired old page, once loaded and used and read, can do for your user? It can preload components needed by the next page, so when the users visit the next page, they have the new scripts, styles and images already in the cache. Next page loads faster and the user's happy. On its death bed you tired old page has left a good inheritance for future generations. Good old page.

How can you go about preloading for the next page? By waiting for onload of the current page and requesting the new components. Here are 4 ways to do so, all using a timeout of 1 second after page load so that prefetching doesn't interfere with the user experience on the page.

One way... (DOM)

Using DOM you can create a new LINK element and a new SCRIPT element and append them to the HEAD. For images - it's a one-liner new Image.src="..."

The drawback of this method is that your CSS is executed against the current page and might affect display. Same for JavaScript - it's executed. The image is simply requested and never displayed.

window.onload = function() {
    setTimeout(function(){

        // reference to <head>
        var head = document.getElementsByTagName('head')[0];

        // a new CSS
        var css = document.createElement('link');
        css.type = "text/css";
        css.rel = "stylesheet";
        css.href = "new.css";

        // a new JS
        var js = document.createElement("script");
        js.type = "text/javascript";
        js.src = "new.js";

        // preload JS and CSS
        head.appendChild(css);
        head.appendChild(js);

        // preload image
        new Image().src = "new.png";

    }, 1000);
};

DOM test page

For this way of doing it you can use any JavaScript library's helper methods to load stuff on demand. Good examples - YUI Get and LazyLoad

... or another ... (using iframe)

Another option is to create an iframe and append your components to its head. Using an iframe you can avoid the CSS potentially affecting the current page. JavaScript will still be executed.

window.onload = function() {
    setTimeout(function(){

        // create new iframe
        var iframe = document.createElement('iframe');
        iframe.setAttribute("width", "0");
        iframe.setAttribute("height", "0");
        iframe.setAttribute("frameborder", "0");
        iframe.setAttribute("name", "preload");
        iframe.id = "preload";
        iframe.src = "about:blank";
        document.body.appendChild(iframe);

        // gymnastics to get reference to the iframe document
        iframe = document.all ? document.all.preload.contentWindow : window.frames.preload;
        var doc = iframe.document;
        doc.open(); doc.writeln("<html><body></body></html>"); doc.close(); 

        // create CSS
        var css = doc.createElement('link');
        css.type = "text/css";
        css.rel = "stylesheet";
        css.href = "new.css";

        // create JS
        var js = doc.createElement("script");
        js.type = "text/javascript";
        js.src = "new.js";

        // preload CSS and JS
        doc.body.appendChild(css);
        doc.body.appendChild(js);

        // preload IMG
        new Image().src = "new.png";

    }, 1000);
};

IFRAME test page

... I'm gonna find ya ... (static page in iframe)

If your components are static you can create a page that has them all and load that page into the dynamic iframe. Static means knowing them in advance, not relying on page's JavaScript to figure them out on the fly. As you can see, much simpler than the previous code.

window.onload = function() {
    setTimeout(function(){

        // create a new frame and point to the URL of the static
        // page that has all components to preload
        var iframe = document.createElement('iframe');
        iframe.setAttribute("width", "0");
        iframe.setAttribute("height", "0");
        iframe.setAttribute("frameborder", "0");
        iframe.src = "preloader.html";
        document.body.appendChild(iframe);

    }, 1000);
};

IFRAME static page test

... I'm gonna GETcha, GETcha, GETcha! (with Ajax)

Finally - Ajax. Scripts are not executed, CSS not used for rendering. Nice and clean. For simplicty the code has no support for browsers that don't know what XMLHttpRequest is.

window.onload = function() {
    setTimeout(function(){

        // XHR to request a JS and a CSS
        var xhr = new XMLHttpRequest();
        xhr.open('GET', 'new.js');
        xhr.send('');
        xhr = new XMLHttpRequest();
        xhr.open('GET', 'new.css');
        xhr.send('');

        // preload image
        new Image().src = "new.png";

    }, 1000);
};

Ajax test page

Thanks!

Any other ways you can think of?

 

Installing a bunch of PNG tools on the Mac

Friday, June 12th, 2009

This is one of those note-to-self type of posts. Just went through the exercise of installing a number of PNG tools on the Mac and here are some notes. The instructions below should probably work on any unix box.

AdvDef, AdvPng, ...

There is a number of Adv* tools (advdef, advpng, advmng, advzip) packed together as AdvComp. Installation difficulty: fairly straighforward.

Download:

$ curl http://softlayer.dl.sourceforge.net/sourceforge/advancemame/advancecomp-1.15.tar.gz \
         > advcomp.tar.gz

Uncompress:

$ tar -xzvf advcomp.tar.gz

Compile and install:

$ cd advancecomp-1.15/
$ sudo ./configure
$ sudo make install

Test:

$ advdef
advancecomp v1.15 by Andrea Mazzoleni
Usage: advpng [options] [FILES...]

Modes:
  -z, --recompress      Recompress the specified files
Options:
  -0, --shrink-store    Don't compress
  -1, --shrink-fast     Compress fast
  -2, --shrink-normal   Compress normal
  -3, --shrink-extra    Compress extra
  -4, --shrink-insane   Compress extreme
  -f, --force           Force the new file also if it's bigger
  -q, --quiet           Don't print on the console
  -h, --help            Help of the program
  -V, --version         Version of the program

Excellent! Next.

OptiPng

OptiPng is another easy install.

Download and decompress:

$ curl http://superb-west.dl.sourceforge.net/sourceforge/optipng/optipng-0.6.3.tar.gz \
            > optipng.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1035k  100 1035k    0     0   171k      0  0:00:06  0:00:06 --:--:--  196k
$ tar -xzvf optipng.tar.gz

Compile and install:

$ cd optipng-0.6.3
$ sudo ./configure
$ sudo make install

Test:

$ optipng
OptiPNG 0.6.3: Advanced PNG optimizer.
Copyright (C) 2001-2009 Cosmin Truta.

Synopsis:
    optipng [options] files ...
Files:
    Image files of type: PNG, BMP, GIF, PNM or TIFF
Basic options:
    -?, -h, -help	show the extended help
    -o <level>		optimization level (0-7)		default 2
    -v			verbose mode / show copyright and version info
Examples:
    optipng file.png			(default speed)
    optipng -o5 file.png		(moderately slow)
    optipng -o7 file.png		(very slow)
Type "optipng -h" for extended help.

Beauty! Next - pngout.

PNGout

PNGOut's source is not distributed openly. But there are binaries for a number of platforms here.

Download:

$ curl http://static.jonof.id.au/dl/kenutils/pngout-20070430-darwin.tar.gz \
         > pngout.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 97597  100 97597    0     0  46704      0  0:00:02  0:00:02 --:--:-- 56739
$ tar -xzvf pngout.tar.gz

This way you end up with a binary named pngout-darwin. Rename and move somewhere where executables live:

$ sudo mv pngout-darwin /usr/bin/pngout 

Test:

$ pngout
PNGOUT [In:{PNG,JPG,GIF,TGA,PCX,BMP}] (Out:PNG) (options...)        Apr 30 2007
by Ken Silverman (http://advsys.net/ken)
Mac port assistance by Jonathon Fowler (http://jonof.edgenetwork.org/pngout)
PNGOUT optimizes PNG size losslessly using my own deflate algorithm (not Zlib)
With the right options, it can often beat other programs by 5-10%. Options:
   -c# PNG output color type: 0=Gray, 2=RGB, 3=Pal, 4=Gray+Alpha, 6=RGB+Alpha
   -f# PNG output filter...

PngRewrite

PNGRewrite was a little trickier, until I realized I need to install libpng first.

Download and install libpng:

$ curl ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng-1.2.37.tar.gz \
         > libpng.tar.gz
$ tar -xzvf libpng.tar.gz
$ cd libpng-1.2.37/
$ sudo ./configure
$ sudo make install

Download, unzip pngrewrite:

$ curl http://entropymine.com/jason/pngrewrite/pngrewrite-1.3.0.zip \
         > pngrewrite.zip
$ unzip pngrewrite.zip

Compile pngrewrite (the make file didn't work for me) and copy the binary where executables are comfortable.

$ gcc -lpng pngrewrite.c -o pngrewrite
$ sudo cp pngrewrite /usr/bin/

Test:

$ pngrewrite
pngrewrite v1.3.0: PNG image palette optimizer
Usage: pngrewrite infile.png outfile.png

That's all, folks

Installing PNGCrush? Blogged before.

And if anyone has an idea how to get deflopt installed, please comment.

 

MHTML - when you need data: URIs in IE7 and under

Friday, April 10th, 2009

In the previous post I described what data: URIs are and how they are useful to reduce the number of HTTP requests. Now, the problem with data: URIs is that they are not supported by IE < 8. But, thanks to this article (in Russian), here's a way to work around that limitation: using MHTML for IE7 and under.

MHTML-a-what?

MHTML is MIME HTML, or if you insist on me spelling it out completely is like "Multipurpose Internet Mail Extensions HyperText Markup Language". In short it's HTML but like email with attachments. In one "multipart" email you can have several... hm, things - HTML version of the email, text-only version, attachment, another attachment...

MHTML is the same but for HTML. In one file you put a bunch of stuff (e.g. image files) and you save on the precious HTTP requests.

MHTML example

Let's check out an example. The HTML is actually not important, it's the CSS where we'll stick all our inline images. Once with data: sheme for all normal browsers and once with the mhtml: scheme for IE before 8.

The overall "template" would look like so:

/*
Content-Type: multipart/related; boundary="_ANY_SEPARATOR"

--_ANY_SEPARATOR
Content-Location:somestring
Content-Transfer-Encoding:base64

iVBORw0KGgoAAAANSUhEUgAAAAQAAAADCAIAA[...snip...]SuQmCC
*/

#myid {
  /* normal browsers */
  background-image: url("data:image/png;base64,iVBORw0[...snip...]");
  /* IE < 8 targeted with the star hack */
  *background-image: url(mhtml:http://phpied.com/mhtml.css!somestring);
}

The multipart stuff goes into a CSS comment. You then use data: URI scheme for all normal browsers. Then you use the star hack to target IE browsers before 8. For these browsers you give the URL of the CSS, exclamation and then a string that uniquely identifies the desired "part" in the multipart comment. Simple, eh? Didn't think so, but hey, it's not exactly rocket science.

Demo example with more parts

Check out a demo page that has two "parts" in the multipart MHTML comment.

The HTML is noting special and the CSS goes like:

/*
Content-Type: multipart/related; boundary="_ANY_STRING_WILL_DO_AS_A_SEPARATOR"

--_ANY_STRING_WILL_DO_AS_A_SEPARATOR
Content-Location:locoloco
Content-Transfer-Encoding:base64

iVBORw0KGgoAAAANSUhEUgAAAAQAAAADCAIAAAA7ljmRAAAAGElEQVQIW2P4DwcMDAxAfBvMAhEQMYgcACEHG8ELxtbPAAAAAElFTkSuQmCC
--_ANY_STRING_WILL_DO_AS_A_SEPARATOR
Content-Location:polloloco
Content-Transfer-Encoding:base64

iVBORw0KGgoAAAANSUhEUgAAABkAAAAUBAMAAACKWYuOAAAAMFBMVEX///92dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnYvD4PNAAAAD3RSTlMAACTkfhvbh3iEewTtxBIFliR3AAAAUklEQVQY02NgIBMwijgKCgrAef5fkHnz/y9E4kn+/4XEE6z/34jEE///A4knev7zAwQv7L8RQk40/7MiggeUQpjJff+zIpINykbIbhFSROIRDQAWUhW2oXLWAQAAAABJRU5ErkJggg==
*/

#test1 {
  background-image: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAQAAAADCAIAAAA7ljmRAAAAGElEQVQIW2P4DwcMDAxAfBvMAhEQMYgcACEHG8ELxtbPAAAAAElFTkSuQmCC"); /* normal */
  *background-image: url(mhtml:http://phpied.com/files/mhtml/mhtml.css!locoloco); /* IE < 8 */
}

#test2 {
  background-image: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABkAAAAUBAMAAACKWYuOAAAAMFBMVEX///92dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnYvD4PNAAAAD3RSTlMAACTkfhvbh3iEewTtxBIFliR3AAAAUklEQVQY02NgIBMwijgKCgrAef5fkHnz/y9E4kn+/4XEE6z/34jEE///A4knev7zAwQv7L8RQk40/7MiggeUQpjJff+zIpINykbIbhFSROIRDQAWUhW2oXLWAQAAAABJRU5ErkJggg=="); /* normal */
  *background-image: url(mhtml:http://phpied.com/files/mhtml/mhtml.css!polloloco); /* IE < 8 */
}

div {
  width: 100px;
  height: 100px;
  font: bold 24px Arial;
}

Drawback

In addition to the drawbacks of data: URIs (bigger CSS, small CSS change invalidates your cached inline images), this method has the obvious drawback that the inline images are repeated twice. But sometimes... a person's gotta do what a person's gotta do ;)

 

data:urls - what are they and how to use them

Friday, April 10th, 2009

If you follow this blog you already know the infamous website performance rule #1 - reduce the number of HTTP requests. Actually, to celebrate Earth Day and to jump the "go-green" wagon/jargon, my favourite performance mantra as of late is "Reduce, Reuse, Recycle" (the Recycle part is a wee fuzzy but, oh well)

So to reduce the number of requests for JavaScript files, you combine all .js into one monolithic file.

Same for .css

For images - use CSS sprites to create one image files that contains all your little icons.

Yet another way to minimize the number of HTTP requests is to use data URLs (proper name is data URI scheme).

"data... uri? urgh-i? An example, please!"

Say you have this minuscule image:

which you include like:

<img src="http://phpied.com/images/check.png" />

This way you're actually using the http URI scheme.

Same thing using data URI will look like:

And the code for it:

<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAQAAAADCAIAAAA7ljmRAAAAGElEQVQIW2P4DwcMDAxAfBvMAhEQMYgcACEHG8ELxtbPAAAAAElFTkSuQmCC" />

This way the image in inlined in the HTML and there is no extra HTTP request to retrieve it.

data URI syntax

Let's take a look again at this img tag and its slightly disturbing syntax:

<img src="data:image/png;base64,iVBOR..." />

You have:

  • data - name of the scheme
  • image/png - content type
  • base64 - the type of encoding used to encode the data
  • iVBOR... - the encoded data
  • some , and ; and : sprinked for good measure

How do I base64-encode?

While there are some online tools that will let you upload an image or point to a URL, you can simply do it from PHP and the command line.

Say you have the filename check.png. Then go:

$ php -r "echo base64_encode(file_get_contents('check.png'));"

This will spit out encoded content for copy-pasting pleasure. Naturally you can also do this from your application code, if needed.

Using data URLs in CSS

If you want the same image as a repeating CSS background, you can use it like this:

<div id="data-uri-test"></div>
<style type="text/css">
#data-uri-test {
  width: 20px;
  height: 20px;
  background-image: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAQAAAADCAIAAAA7ljmRAAAAGElEQVQIW2P4DwcMDAxAfBvMAhEQMYgcACEHG8ELxtbPAAAAAElFTkSuQmCC");
}
</style>

Where the gist of it really boils down to the same:

background-image: url("data:image/png;base64,iVBOR...");

Lights, camera... IE!

Everybody's favourite part of every technique... IE support. The thing is IE8 supports data URIs. Earlier IEs do not. But, there is a solution involving MHTML, which I'll describe in a follow-up post, since this one turn out kinda longish.

For the impatient and Russian-speaking among you - click here.

 

Picassa’s progressive photo rendering

Thursday, March 5th, 2009

If you've followed the series of image optimization posts on the YUIblog, you've probably seen the one about progressive JPEGs. In short, if a photo is over 10K it has a high probability of being smaller in file size when you use progressive JPEG.

How do you turn a normal, baseline JPEG into a progressive one? Well, most image software has this capability. On the command line you can use JPEGTRAN's -progressive flag.

Progressive JPEGs don't render progressively in IE but that's not a big deal, they still render.

Mi casa, Pi-cassa

That's all nice and good, but as I was browsing Picassa, I noticed some serious progressive photo rendering going on. Imagine my surprise when I found the photos on picassa seemed to be rendering progressively even in IE. And upon inspecting one photo - it wasn't even using progressive encoding!

small thumb first, bigger image later

Picassa uses a clever trick where they quickly load a small thumbnail of the photo you want to see, and then start loading the real big image. This way you get something that gives you an idea of the image and if you don't like it, you can click next and move on without loading for the whole image. And what gives the feeling of progressive loading is the fact that the thumbnail is stretched to take the whole width/height of the real image.

See for yourself

This page initially shows a small 128x96 image:
small thumb

But this image is stretched to 640x480 so it looks like:
small thumb stretched

The actual final image is
big image

This is pretty trivial to achieve, something like (not tested, just typing away):

var i = new Image();
i.onload = function() {
    document.getElementById('small').src = i.src;
}
i.src = "big.jpg";

So once the big image arrives, it replaces the small one and the visual effect is like progressive rendering - as if the image increases quality gradually.

 

Hide broken images

Monday, February 16th, 2009

I know, you don't have broken images on your site, it's unprofessional and ugly. But sometimes you may be loading images that you don't control and you never know what's going on on the other server you're expecting to serve, but it may not feel up to the task.

One nice and simple strategy to deal with this uncertainty is to hide the images that fail to load. Browsers sent an "error" event when the worst happens and an image fails for whatever reason. Subscribe to this event using your favorite event-listener-attaching approach or library and hide the image.

Or with some ugly old-school inline event handler:

<img
  src="broken.png"
  onerror="this.style.display='none'"
/>

Simple, eh?

 

Installing JPEGTRAN on a Mac or Unix/Linux

Friday, January 16th, 2009

JPEGtran is cool because it lets you optimize JPEG images losslessly by:

  1. Stripping meta data (meta is sometimes bulky and useless for web display)
  2. Optimizing Huffman tables or
  3. Convert a JPEG to progressive encoding

From my experience 1 is more important than 2 or 3 and 3 gives better results than 2 for images over 10K

Installation

I never had to install jpegtran before because all unix/linux machines I've touched already have it. And on windows you just copy a binary somewhere in your path.

Well, I got this MacBook now and it doesn't have jpegtran so had to figure it out myself. Here's how you can do it, worked for me on Mac OS should work on any unix/linux too.

BTW, jpegtran is part of a package of few tools known as libjpeg, so you'll be installing a few programs not only jpegtran.

  1. Get the source code from here. It's the file called jpegsrc.v6b.tar.gz. Using cURL you can download like:
    curl http://www.ijg.org/files/jpegsrc.v6b.tar.gz > /tmp/libjpeg.tar.gz
  2. Uncompres the package, e.g. tar -xzvf /tmp/libjpeg.tar.gz
  3. go to the directory that contains the uncompressed code, e.g. cd /tmp/jpeg-6b
  4. ./configure
  5. sudo make install

Done.

You can test your shiny new set of tools like this and get some help information about the various options:

> jpegtran -h
> cjpeg -h
> djpeg -h
> rdjpgcom -h
> wrjpgcom -h

You also test by optimizing my book cover from Amazon like:

curl http://ecx.images-amazon.com/images/I/41ckBp3bBUL._SL500_AA240_.jpg > oojs.jpg
jpegtran -copy none -progressive oojs.jpg > oojs-opt.jpg

This gives you 10% smaller file with not a pixel of quality loss. Not bad, eh, for a minute of work, or less.

 

Image optimization - in Chinese

Thursday, January 8th, 2009

Thanks to Joseph Jiang who translated in Chinese parts of my image optimization articles from the YUI blog

If you read Chinese, visit http://josephj.com/entry.php?id=209.

 

Paint.NET is cool…

Tuesday, December 23rd, 2008

... but doesn't write PNG8 with alpha transparency, unfortunately.

This comment on the YUI blog got me all excited by the possibility of having another designers tool other than Fireworks that creates PNG8 (palette PNGs) with alpha transparency.

Overall Paint.NET is a very simple and friendly program (as a non-designer I'm often intimidated by Photoshop's plethora of features) and I can see how I can use it for my future (rare, I hope) design needs :) It's pretty impressive, free and from what I read, it has a dedicated community, plus a number of plugins (hmm, a smush.it plugin would be nice... one day).

I tried saving a PNG8 with alpha but unfortunately this feature is not (yet!) supported. I created an image with a sold red square and blue gradient that fades to 100% transparency. When saving as 8-bit (in order to create a palette PNG) the semi transparent pixels are all gone. It treats palette PNGs like GIFs (like Photoshop does).

Default (produces truecolor PNG):
png 32

With 8 bit option selected:
png8

 

Installing ExifTool on Dreamhost

Tuesday, December 23rd, 2008

ExifTool looks like a very promising tool to fiddle with all sorts of JPEG metadata (needed for smush.it) but first I had to make sure I can install it on Dreamhost. Although installation didn't go as described on the exiftool site (since I don't have sudo access on Dreamhost), it's still installable and it's actually pretty easy.

  1. ssh to your DH box and go to the directory that will contain the tool, in my case it's my home directory
  2. Download the latest version of the code, e.g.
    wget http://www.sno.phy.queensu.ca/~phil/exiftool/Image-ExifTool-7.59.tar.gz
  3. Uncompress the code archive
    tar -xf Image-ExifTool-7.59.tar.gz
  4. Delete the original archive
    rm Image-ExifTool-7.59.tar.gz
  5. Rename the newly created Image-ExifTool-7.59 to something shorter to type, e.g. "et"
    mv Image-ExifTool-7.59 et
  6. All done! You can now run it from any directory and show help info like
    ~/et/exiftool - h

You can also test the new installation with some of the images found in the exiftool test directory, like so:

$ ~/et/exiftool ~/et/t/images/IPTC-XMP.jpg

ExifTool Version Number         : 7.59
File Name                       : IPTC-XMP.jpg
File Size                       : 20 kB
File Modification Date/Time     : 2005:12:31 13:05:50-08:00
File Type                       : JPEG
MIME Type                       : image/jpeg
JFIF Version                    : 1.02
Exif Byte Order                 : Little-endian (Intel, II)
Image Description               : A witty caption
Make                            : FUJIFILM
Camera Model Name               : FinePix2400Zoom
Orientation                     : Horizontal (normal)
X Resolution                    : 72
Y Resolution                    : 72
Resolution Unit                 : inches
Software                        : Adobe Photoshop 7.0
Modify Date                     : 2004:02:26 09:36:46
Artist                          : Phil Harvey
Y Cb Cr Positioning             : Co-sited
Copyright                       : Copyright 2004 Phil Harvey
F Number                        : 3.5
Exposure Program                : Program AE
ISO                             : 100
Exif Version                    : 0210
Date/Time Original              : 2001:05:19 18:36:41
Create Date                     : 2001:05:19 18:36:41
Components Configuration        : YCbCr
Compressed Bits Per Pixel       : 1.6
...
... [snip]
...
Encoding Process                : Baseline DCT, Huffman coding
Bits Per Sample                 : 8
Color Components                : 3
Y Cb Cr Sub Sampling            : YCbCr4:4:4 (1 1)
Aperture                        : 3.5
Image Size                      : 100x80
Shutter Speed                   : 1/64
Focal Length                    : 6.0 mm
Light Value                     : 9.6
 

PNG optimization tools

Monday, December 22nd, 2008

I'm currently experimenting with different tools for optimizing PNG images to figure out strengths/weaknesses of each. Only considering free, ideally open-source, tools that can be run from the command line. For smush.it I just picked pngcrush for no particluar reason and I was thinking that once I have the optimization tool up and running and get the point across that images can be optimized with no human intervention as part of the normal "push" process... then the actual behind-the-scenes png optimizer can be tweaked without affecting the overall workflow.

So, I'm considering these PNG optimizers:

Can you think of any other PNG optimizers out there? Please comment.

Well, I just came back from a cruise with no internet access ($1/minute is unacceptable!) so I used my laptop's idle time to run some of the tools on a bunch of files (well over 10 000 PNG files, smartly collected right before the cruise ;) ).

So far some (very preliminary!) results from running the tools.

  1. pngcrush's -brute option is slower, as expected and usually doesn't help much
  2. pngcrush with -rem alla (strip all chunks only keeping the transparency chunk) doesn't get much better than -rem none (keep all chunks, even useless ones)
  3. pngout seems to give the best results, but it's probably the slowest
  4. pngrewrite is only for pallete PNGs (PNG8), it also seems to convert truecolor PNGs with less than 256 colors to PNG8

Again, these are very preliminary and again, please let me know if you know of any other PNG optimizers.

And keep an eye on the YUIblog's img opt series for more detailed report of the results.

 

image diff

Saturday, November 15th, 2008

Was having fun today with idiff.php - a PHP shell script to tell you if two images are visually different by comparing them pixel by pixel. If there's a difference, the script creates a third image - black background with the different pixels in green. Only after writing the script I found that there's an imagemagick command compare that does the same :)

the script

The idea is that two GD images are created from the input files, also a third GD image with black background to store the diff. Loop through all the pixels and compare them one by one. If one is different, write a green pixel at the same location of the diff image.

<?php
/**
 * Shell script to tell if two images are identical.
 * If not, a third image is written - black background with the different pixels painted green
 * Code partially inspired by and borrowed from http://pear.php.net/Image_Text test cases
 */

// check if there's enough input
if (empty($argv[1]) || empty($argv[2])) {
    echo 'gimme at least two image filenames, please.', "\n";
    echo 'e.g. "php idiff.php img1.png img2.png"';
    echo "\n", 'third filename is the image diff, optional, default is "diffy.png"';
    exit(1);
}

// create images
$i1 = @imagecreatefromstring(file_get_contents($argv[1]));
$i2 = @imagecreatefromstring(file_get_contents($argv[2]));

// check if we were given garbage
if (!$i1) {
    echo $argv[1] . ' is not a valid image';
    exit(1);
}
if (!$i2) {
    echo $argv[2] . ' is not a valid image';
    exit(1);
}

// dimensions of the first image
$sx1 = imagesx($i1);
$sy1 = imagesy($i1);

// compare dimensions
if ($sx1 !== imagesx($i2) || $sy1 !== imagesy($i2)) {
    echo "The images are not even the same size";
    exit(1);
}

// create a diff image
$diffi = imagecreatetruecolor($sx1, $sy1);
$green = imagecolorallocate($diffi, 0, 255, 0);
imagefill($diffi, 0, 0, imagecolorallocate($diffi, 0, 0, 0));

// increment this counter when encountering a pixel diff
$different_pixels = 0;

// loop x and y
for ($x = 0; $x < $sx1; $x++) {
    for ($y = 0; $y < $sy1; $y++) {

        $rgb1 = imagecolorat($i1, $x, $y);
        $pix1 = imagecolorsforindex($i1, $rgb1);

        $rgb2 = imagecolorat($i2, $x, $y);
        $pix2 = imagecolorsforindex($i2, $rgb2);

        if ($pix1 !== $pix2) { // different pixel
            // increment and paint in the diff image
            $different_pixels++;
            imagesetpixel($diffi, $x, $y, $green);
        }

    }
}

if (!$different_pixels) {
    echo "Image is the same";
    exit(0);
} else {
    if (empty($argv[3])) {
        $argv[3] = 'diffy.png'; // default result filename
    }
    imagepng($diffi, $argv[3]);
    $total = $sx1 * $sy1;
    echo "$different_pixels/$total different pixels, or ", number_format(100 * $different_pixels / $total, 2), '%';
    exit(1);
}
?>

usage

Type in the command line something like:

php idiff.php img1.png img2.png result.png

This command will compare img1.png with img2.png. If they are not the same dimensions, will exit with error code. If their pixels exactly the same will exit with success code 0 and print a success message. If the images are not the same, will write the file result.png.

If you omit result.png, the generated file will be called diffy.png

example

img1.png:

img2.png

running the command:

>php idiff.php img1.png img2.png
70/24600 different pixels, or 0.28%

the result diffy.png:

imagemagick

yep, so after I did this I found that imagemagick has a command called compare that does the same, only with more features, like the -fuzz option for example that allows you to specify how different should the pixels be, in order to consider them important.

the command:

>compare img1.png img2.png diffy-im.png

the diffy-im.png result

As you can see the different pixels are red and the actual image is a very pale background. Nice.

 

php|works Atlanta

Thursday, November 13th, 2008

Thanks to everyone who attended my image optimization talk at the php|works + PyWorks conference in Atlanta. And thanks for all the questions! I love questions, feels more natural - just geeks talking to geeks - as opposed to one guy sitting on a podium and talking.

And the slides:

 

smush.it update

Sunday, October 26th, 2008

What's new in smush.it?

  • The old domain smushit.com now redirects to the new one smush.it, which is what we originally intended but the domain registration took a while and we quickly got smushit.com just in time for the Ajax Experience announcement of the tool
  • There's a bookmarklet version of the Firefox extension so you can now run in IE or any other browser capable of bookmarking
  • There's an FAQ
  • The API, while not really documented, is at least described in the FAQ
  • So is the privacy policy. People said we should have privacy policy, we're not lawyers, so we couldn't write something that sounds important. The policy is basically: 1) we won't use the images you upload in any way other than gathering aggregated optimization statistics (how much we save on average, how many files get smushed daily...). 2) Images will be deleted from the server, once we get around to writing some sort of stats script, so download the optimized zip asap, don't rely on it being there for long. 3) The images you upload are available to anyone who can guess the random URL smush.it creates, so don't upload images you want to keep private
  • Flash 10 support, updated to YUI 2.6's flash uploader and sweated a lot to make the uploader work in the tightened flash 10 security, but that's a whole other blog post
  • Smu contest! What's a Smu? We want to know too. Send us you idea of a Smu to smu at smush.it