While you are shifting up through the gears there is no problem or sound, but if take your foot off the throttle there is a grinding sound from the gearbox near the passenger footwell, I'm thinking it could be front diff problem, anyone else has encountered this problem? any help would be appreciated.

Thrust bearing on the "output" side of the gearbox would be my guess. While the car is under acceleration or load its putting pressure on the bearing and without the acceleration (load) there's no pressure on the thrust bearing.

That would be good if it was just the thrust bearing.

Thanks