Yeah, you can do that with gmail, but "doesn't keep a copy" is a technicality. They don't guarantee that they freed up or overwrote the space, just that if you open the web interface it won't be there. So, convenient if you prefer your own email client, not convenient if you don't trust google.