According to the 1850 census, there was a Joseph Hollenbeck living in Athens, Greene County, New York, who was born about 1832. I believe he was living on a farm with his father, Jacob, his mother, Mary, and his five siblings. I don't have any evidence to prove that this is our Hollenbeck but it feels right.
The interesting thing about this is that Athens, New York, is just a few miles south of Albany and we have evidence of ball-playing in the region dating back to the 18th century. There was an organized baseball club in Albany by 1856 and it's entirely possible that Hollenbeck could have been exposed to the ball-playing culture of the New York City area while living in Athens. There is no evidence supporting the claims in the secondary sources that Hollenbeck played for the Knickerbocker Club and I completely dismiss that entire idea. But I always believed that, having been born in New York, Hollenbeck was likely exposed to the New York ball-playing culture as a boy and could have brought that knowledge and experience with him to St. Louis. Also, as of right now, we can place Hollenbeck in St. Louis no earlier than 1857 and, therefore, it's possible that he was exposed to the New York game in the greater Albany area in 1856.