It seems like this could slow down display if an external site has a non-robust server and randomly disappears, gets overloaded, loses track of images, etc.

I'd go with a local cache refreshed daily or something.