Instance is a term to describe areas that are generated when a player or group enters them. Many copies of an instance can be run at the same time.
The amount of players per instance is based on peer-to-peer (P2P). On PC there is no hard limitations on how many players can be in one instance. Some managed to have more than 100 players in the same instance. Due to connections and networking reasons the match making system tries to put players in the "best" possible instance for performance. This usually means lower player numbers per instance. The quality of P2P decreases as the amount of players in the same instance increases.
Large groups of players in one instance rarely happens, because most people are spread out across the galaxy.