You can reach everything easily you may want to see in Budapest by public transportation.
Take bus 7 to Kewleti railway station, tram 47 or 49 to Deák Square where all metro lines meet. You can also get Tram #18 which goes to the Castle Hill.
You can even walk to the centre; it is not more than 10 minutes.