Are mushrooms poisonous when gardening?

Hi, I was clearing away leaves in the garden today and I noticed a lot of mushrooms. Some were white and very twisted, then the classic mushrooms and then some that looked brown and white and quite large. I wasn't wearing gloves but I used garden tongs to pack the leaves into bags. And when I was raking the leaves together with my garden rake I broke all the mushrooms – cut them off and smashed them a bit because they're not pretty and they can go straight in the trash. Anyway, I heard that two children had some kind of incident with the death cap mushroom, which is highly poisonous. Now I'm worried that I had it in the garden and broke it and inhaled its smell because I often smell mushrooms. Are the smells of mushrooms also poisonous or only if you actually eat them.