Causeway Bay is one of the busiest part of Hong Kong -- it is a busy district until midnight. You definitely can have a lot of fun there! It is definitely a good choice to book a hostel in Causeway Bay if you want to stay in Hong Kong for a few days; it is very convenient to travel from Causeway Bay to other parts of Hong Kong Island and Kowloon by MTR or bus (Causeway Bay is a station on MTR's Island Line). If you want to go to Kowloon, you may use MTR or the bus stop in Canal Road East. When traveling to your Causeway Bay hostel by MTR, you should check a map and find the nearest MTR station to your chosen hostel before your trip -- the Causeway Bay station may not be the closest one, as Hong Kong is such a small city.
If you like to go shopping, Causeway Bay is a desired place for you. There are some mega department stores and shopping centers in the district, for example, Sogo Department Store and Times Square. However, if you are a budget traveler, don't choose a hostel in the central part of Causeway Bay, because it can be hard to find a fair-priced restaurant or supermarket in that area. There are lots of restaurants in Causeway Bay, serving many types of cuisine, including Chinese, Western, Indian, and Thai -- but do note that these restaurant can be comparatively more expensive than in other parts of Hong Kong.