Hi- My daughter broke her ankle at 2.5 and had a cast on for about 4 weeks. When they took it off, we were told the same -- it will be fine in a few days. It wasn't! It took her a good month or so to get back to normal...maybe more. Now she's 3.5 and I can't even remember which ankle it was that broke! She's absolutely fine. I know it's scary, but I'm sure your son will be ok with time.