It seems like you're exaggerating the problem a bit. Most anime are "inanimate" enough of the time that reading the subtitles won't have resulted in you visually missing anything, aside from a bunch of lip flaps.
On the other hand, if the scene is busy, in my own experience it's just as difficult to catch all of the dialogue by ear as it is by eye. At times, much harder -- to the point that I got into the habit of putting on subtitles for things that are already in English. (Some of the people in my family have caught this bug, too.)
Perhaps it's differences in people's sensory/information processing abilities that result in the variety of viewing preferences that we see...