leaking penetrations

I had a standing seam metal roof installed appropriately 2 years ago, over a a sound built up rolled roofing system. No history of leaks, ever. Within 6 months of installation, I began having leaks at penetrations. No bellows-type boots were used on the penetrations (gas, electric, stand pipe, kitchen stove vent, water). Instead, they were sealed with caulk. The contractor has come out and replaced nearly all the caulk, but I am worried that this problem will reoccur. No flashing was installed around my AC/furnace unit ductwork, that also penetrates, nor around 2 skylights, just caulk. What can / should I do? Thanks.