Unfortunately there isn't currently a way to convert a Public Group into a Private Group.
Your best bet would be to create a new Private Group and invite all the people you'd like to chat with to it from your Public Group.
The same applies to switching a Private Group to a Public Group. Your best to just create a Public Group and invite your Private Group members!
If you see a large influx of people joining your Public Group, it could be because your group falls into a Popular Search category. We often change the Popular Search terms though so your group might not be there for long :)