Most kinds of noise stress me out a lot, and I have a strong preference for silence. I tend to quickly notice quiet environments. Does this occur to anyone else? Any idea why?

Think about a banging sound at 0dB, 1dB, ..., 90dB. Everyone will be bothered at 90dB. Nobody will notice at 0dB. Somewhere in between, it transitions from effortless-to-ignore to impossible-to-ignore. Where is that point? It's not determined by the laws of physics, it's arbitrary, it's a setting in your brain, and it's set differently in different people. Similar for touch. Neurotypical people will not be able to ignore a woodpecker pecking them in the back, but may find it effortless to ignore a shirt tag. Other people find that the shirt tag keeps drawing their attention, but a gentle enough touch sensation would not rise to attention.

The "intense world theory of autism" (about which I'll finish a blog post draft one of these years...) says that autism happens when empathetic social interactions are so overwhelming that the person learns early in life to just avoid thoughts and situations that solicit those feelings, including deliberately avoiding eye contact etc. Not coincidentally, people with autism are liable to have sensory sensitivity too, i.e. to feel overwhelmed by levels of sound and touch that are far lower than what it takes to overwhelm neurotypical people.

Anyway, long story short, we all have innate reactions to different stimuli [LW · GW], and the thresholds can be set differently. That's just the way it is, I think.

Well ADHD is comorbid/associated with stuff like that. Makes sense, ADHD is a specific kind of brain damage in five specific regions. Effects of brain damage can be diverse and random. [in addition to the more common predictable ADHD effects]

As to why? Nature is lazy and your brain is "good enough" to exist as is, even if the internal wiring is a mess.
[there is probably a better answer in the linked wiki article though]

Very rarely I feel what you feel, and what I believe to some kind of underlying sensory processing disorder. 
When I didn't get enough sleep, noises can become overwhelming and music at the gym is so distressing all of a sudden, that I leave immediately. Normally I just avoid loud environments and can't concentrate with laptop fan noise (if the frequency is too high or it's too loud). 
And I just dislike music (but can tolerate) music from Bluetooth speakers.
 

Probably being semi-deaf whilst a toddler made something not calibrate right.
But instead, I just live with constant songs stuck in my head, constant running mental dialog or daydreams. 
With the inability to really filter, parts of my brain adapted by trying to be louder than the noise. [these days much it's less bad, with stimulants for the ADHD and loads of meditation over the years]

All my life. Personally, I blame it for the high anxiety level, but never interested in details. This may provide some leads: https://www.bbc.com/worklife/article/20191115-office-noise-acceptable-levels-personality-type