Thanks for the replies, so update, today I tried messing with the choke, I used a paperclip to hold it in place (I'm thinking it could be an issue with the auto choke system), which fixed the problem where it would idle too high, but it still intermittently drops down.
It does have a fast idle screw and an idle speed screw, but it was working perfectly before and I haven't touched them, so i don't think those could be the issue.
When it does idle too low, holding the choke slightly closed brings it back up. Could be an issue with the fast idle cam.
One of my other ideas is that it could be a vacuum leak in one of the vacuum systems, like the hot idle compensator, so the vacuum leak only happens when that system is active, but this seems unlikely.
If anyone has other ideas, I just wanted to check all other possibilities before I do a carb rebuild.
Thanks,