Hi Michael,
Thanks for taking the time to provide this suggestion.
You could certainly implement a check in your software to count the number of clients in the CMS and check that users are on the appropriate productid or productname (from the licence check data) for that client number.
Your software could then be coded to behave however you desired when the limit was reached. In WHMCS we chose to make clients above the client limits inaccessible; you might prefer to display a licence error or something else entirely.
One option to automatically upgrade customers to the next tier when a certain threshold is reached, could be done using the UpgradeProduct API call: https://developers.whmcs.com/api-reference/upgradeproduct/
You have the serviceid in the licence check data, so can pass this to the call, along with the desired new productid for the higher tier.
Using the API Credentials feature it would be possible to create a credential pair with only the access to use the upgrade function, so there's no concern about opening full access to your installation's API from remote locations.
I welcome further discussion of this in our community: https://whmcs.community/forum/41-developer-corner/