Disappointment with voice recognition

My initial playing around with Microsoft's Speech API was quite successful... its speech recognition accuracy seemed very solid, even if I took my headset off and rested it beside the monitor, or put in on the ground. (Wow!) That was until I tried using it in dictation mode, which increases its vocabulary from a few words to a few thousand words. Suddenly it was recognizing a short sentence correctly with < 5% accuracy.

I figured by reducing its vocabulary to say 150 words I'd get the accuracy back, but I couldn't find any easy way to do that. As a work around, I created a custom grammar that supports statements consisting of 1, 2, 3, 4, 5, or 6 words, each word needing to come from a list of 150 possibilities. Even then its accuracy was remarkably poor. For example, here is what it recognizes "My name is Daniel" as on five different attempts:

"My name is Daniel" was recognized as:

my name is Daniel
I name is Daniel
my name is Daniel
nineteen's Daniel

"I live in Waterloo" was recognized as:

"mon"
"man"
"I man name Waterloo"
"Bigham eighteen mon"
"mon"
"I live name one million"
...

In other words, about 50% in the best cases, and < 10% in many other cases. I would have hoped that the accuracy would have been at least 95% with such a small vocabulary.