There is a CheckSerial function, which accepts the serial key string as a parameter. You can create any algorithm you want to verify it and set the boolean result accordingly.

Basically you define the CheckSerial function in [Code] section of the script. Then you need to check if the entered serial is correct - you need to find your own secure algorihtm for that - Giving you one would be pointless for security :-) This way you can generate multiple keys that will fit the algorithm and pass your check.
And depending on that check you set Result to True or False - Inno Setup wizard will accept or reject entered serial depending on that.

There are many code examples at stackoverflow.com if you want to get a general hang of the coding system. Also Inno Setup has very good documentation online.

I never needed a way to generate serial codes (no-DRM after all :-) ), so I don't know much about it.

Check this article. Code examples are in Delphi, so it should be easy to implement them in IS :-)