It's not like Esperanto at all actually. To quote Wikipedia: "There is no consensus on what International Sign is exactly. It may either refer to the way strangers sign with each other when they lack a common sign language, or it can refer to a conventionalized form used by a group of people with regular contact."
Because iconicity (signs that look like what they mean - the sign language equivalent to onomatopoeia in spoken languages) is so common in sign languages, it's much easier for users of different languages to communicate. Something like 30% to 50% of vocabulary in sign languages is iconic, so it's much easier to guess at the meanings of words from someone else's language.
It's like if the delegates showed up to the first UN meeting and tried to communicate like this: https://old.reddit.com/r/TikTokCringe/comments/1vmqc2i/peppa...
Speakers are prioritizing words and grammar they think will be understood by their conversation partners. Then, over time, people started to standardize, and in some cases write down, which words were best used in that context.