Reusing jars and other containers is frugal, but if you don't think a label stating pickles or spaghetti sauce is stylish, you'll want to remove them. Soaking some labels in plain water will help remove them, but others use a different type of glue that just won't come off with water. These will come off with lamp oil. Lamp oil is refined kerosene (you can use kerosene, since it's cheaper). Either one will dissolve gummy glues. Put it on with a small cloth, then leave the cloth on it until you can rub the label off.
Edited to add: You can also use nail polish remover.