Our 1st Noise Complaint

So its finally happened. The neighbors below us complained that we are too loud. Particularly in the mornings. I'm sure they mean ALL the time... but they chose the morning as their main complaint. Bleh. The people below us are really nice. We've spoken to them on several occassions.. John has helped them put up curtains, I've flooded their kitchen, they've given us coffee. Its a good, neighborly relationship. Except we are now official noisemakers. Booooo.
I feel bad. Moreso, now i'm super paranoid about every single move I make...thinking can they hear this? But at the same time, what am I supposed to do? Khai is ONE.. one year olds make noise. Kids make noise. We live in a condo. The floors are shit. The carpets are shit. The padding needs replacing. We are broke. So.
I asked John what they said exactly when he saw them. He said he was walking in the hallway and they just randomly blurted out You guys are noisy! But he said they said it nicely.
He said he was sorry and that we try to keep it down as best as possible. He said the wife told him its mainly in the mornings that she can hear everything we are doing. He tells her Khai wakes up super early.
I have the rubber alphabet mat in the living room - of course for Khai to play on - but also as a semi padding... hoping it would buffer any noise. Maybe I should get another one for the bedroom. I am also going to have to take his walker away from him in the mornings and his popcorn popper pusher thing. What the hell do you call those damn things anyways.
I'm wondering if her complaint stems from many mornings worth of noise, or this weekend when Khai dropped all the stuff into the tub, including the popcorn popper thingy.... and me roaring and carrying on with him. I don't know. Bah. To top it off, the neighbors down the hall, THEIR kids are super noisy. Running up and down the halls at all hours. And now i'm all paranoid that the neighbors below thing their noise is our noise!
I hate living here.