I build muscle really fast

Genetic factors are certainly at play when it comes to muscular potential as well as athletic performance and longevity. No one knows all the factors yet, but you're genetics dictate hormone production, points of insertion between muscular and skeletal systems (leverages), percentage of fast vs slow twitch muscles (explosiveness vs. endurance of muscle contractions), and many other things.

Hormonal profile can adversely affect your ability to recover from physical stresses such as the the "micro-tears" you mentioned as well as stress on the central nervous system. This is why steroids are so powerful (they dramatically alter the hormonal profile); their most valuable feature is the ability to accelerate recovery, which leads to faster and bigger muscle gain. Steroids are often used on patients with traumatic injuries such as from car accidents and bad falls to quickly get them back on their feet.

The length of your limbs and where your muscle bellies attach to your bones affects the leverages of you movements. Certain body proportions are more advantageous at generating force than others (eg. shorter arms and tricep attachments closer to the elbow joint make for a stronger bench presser and bicep attachment closer to the elbow joint lead to better bicep peak size development).

Freakishly fast and powerful athletes such as those in the NFL and NBA are all naturally explosive, which is something you can't really train to a large extent (although training can bring out any natural strengths one might have). They have a higher ratio of fast to slow twitch fibers, which can generate quick and violent bursts of power (eg. the 40-yard dash is such a short distance that acceleration trumps top speed in getting a great time. NFL players normally run a sub 5-sec 40, which is wicked fast acceleration made possible only by natural talent AND solid training).

So the short answer to your question is an emphatic YES. While we would all like to believe that hard work trumps everything, physical/athletic potential is dictated largely by your genetics, and that potential can realized through hard work and proper training/discipline. Steroids and other PED's are X-factors that often complicate this viewpoint (and is worth its own discussion elsewhere), but most athletes ostensibly stay away from such things.

EDIT: After reviewing other answers here, somatotypes were brought up. This concept is archaic and misleading by modern standards (most people do not fall squarely into any of the 3 traditional somatotypes and numerous treatments of this topic have been published discrediting it's generalization of body-types), but does support the idea that some individuals are more muscle-bound than others.