Awkward elevator chit chat

We use the elevator in our building almost every day.  Most of the time we ride alone, but sometimes we share with another couple, a group of kids, or the lone old man or woman.  Sometimes we ride in silence or talk to our respective groups, sometimes smile at one another, or my favorite - share cheerful "ni haos".  Every once in awhile someone tries to strike up a conversation in Chinese.  Normally we apologetically say in English "no Chinese".  Last time, however, I wasn't thinking and said it in Chinese.  Well, I tried to say it.  In the moment I got flustered and instead said either "You don't speak Chinese?" or "I don't speak Chinese?" (I can't really remember what I started with because I was so focused on the fact that I had accidentally turned the statement into a question).  The old man laughed at me.

We're trying out Chinese lessons this week.  Maybe next time I ride with him I will be able to correctly put together a statement.  Or maybe I should just stick with ni hao.