Below are some things that may be useful to bear in mind when designing special abilities for monsters.
Categorization of abilities by their uniqueness
Fans of RPG Superstar probably know the terms "spell in a can" and "Swiss army knife", which are used to describe certain types of magic items. The former means you only re-use a spell's effect instead of creating something new, while the latter refers to an arbitrary collection of effects with no thematic link, let alone mechanical synergies.
I think monster special abilities and their uniqueness (unique > derivative > re-flavored > re-used) are analogous with the effects of a wondrous item: re-used abilities are the equivalent of a SIAC, re-flavored and derivative abilities are pretty much the same as a SIAC with a twist, while unique abilities are similar to unique effects in items.
Abilities from the universal monster rules: Abilities such as all-around vision are listed in the universal monster rules section of the Bestiary. The good thing about UMR is that you don't need to include the full rules text for an ability borrowed from there.
Re-used abilities: Some abilities are fairly common (e.g. death throes), but they don't appear in the UMR.
Re-flavored abilities: Re-flavoring means you don't change the mechanics of an ability much (or at all), but you dress it up as something new. For example, a fey living in the woods might be able to call upon the roots in the ground to strangle her enemies, using black tentacles with a few twists as the mechanics for it.
I like re-flavoring, but I think in a contest you shouldn't rely on re-flavoring alone. It doesn't test your rules-fu quite so much as building an ability from the ground up. Of course, with very creative re-flavoring it's possible to create effects that feel unique but only take up a few words.
Derivative abilities: While the exact details may be your own, you borrowed the idea from an existing ability. Same as with re-flavored abilities, derivative abilities are good when used in moderation. They share the same problem; it's harder to show off your skills and mojo if you only swap out a few words. A few examples of derivative abilities can be found in my mock entry ”obsidiapteryx”. Death throes, yep, mostly re-used text with minor mechanical changes. Magma bomb, yes, slightly altered and fe-flavored alchemist's bomb. Razor feathers, yep.
Unique abilities: These are abilities that don't borrow rules text from other abilities. As such, they're the most creative type of abilities. I think it's always a good idea to include at least one, but preferably two or three of these if you're designing a contest entry. Although it's new rules text, you should imitate the Paizo style in how you structure the abilities and how you use terminology, etc. When you use standard phrases, write them exactly as they appear in other monster entries.
Of course, creating unique abilities isn't everything that matters. Abilities that are less creative in and of themselves may be creative in the context they appear in, i.e. how they fit the monster concept. Conversely, designing creative abilities doesn't guarantee that they make sense for the creature to have.
How many abilities?
I'd say, the recommended maximum number of new special abilities (i.e. not counting those in the universal monster rules, only those listed as special abilities) is five. Three is probably the sweet spot. One is usually too few; it's hard to tell how good your rules-fu is by looking at just one ability.
It's also a good idea to keep the total number of special abilities in check. The more special abilities a monster has, the harder it is to keep track of everything, and playing the monster may become a chore. And it probably won't get a chance to use them all anyway.
The mechanical complexity of the abilities
Much like magic item effects, monster abilities should be fairly simple and straightforward. Don't include more die rolls or math than is necessary. One of the biggest challenges in monster design is to create abilities that are fun and innovative, and challenge the players and PCs in interesting ways without introducing too much mechanical complexity. An ability that gives the monster a +4 bonus on a stat, for example, is definitely simple, but it's also very boring.
Some things to avoid
Below are some things I came across in Here Be Monsters entries that I'd recommend against.
- Abilities that make it practically impossible for the PCs to do anything or even notice the creature, such as a combination of incorporeality and invisibility.
- Abilities that only work under very specific circumstances or only against a certain type of enemy. Things like these limit the GM's options. It will be harder to place the monster in an encounter or adventure, or at least they make the encounter less interesting if the specific circumstance doesn't occur. Things like these might also turn the monster into a plot device: if the monster can only do one thing, it'll always be used for just that one thing.
- Abilities that aren't thematically connected to the monster concept. If a ”blood reaper” has a ”fire mantle” ability, there should be a good reason for it. The more focused the theme and concept are, the less you need to explain it for it to make sense.
- Abilities that target males or females specifically. Especially if the reasoning is that members of these genders/sexes should find the monster sexually attractive. There are a lot of unfortunate implications there–not all characters are straight. Targeting a creature's sexual preference is also problematic as it's something players generally haven't written down on their character sheets.
- Leaving something up to GM discretion is generally not a good idea. Too much GM discretion can grind a game to a halt, or cause arguments, especially with less experienced GMs. Also, it's good to remember there are campaigns like Pathfinder Society, where table variance should be avoided.
- Similarly, mechanically vague abilities (e.g. knowing a target's ”deepest desire”) are problematic. Sometimes authors realize an ability is too vague to be useful and they write a list of some kind to make up for it. Sometimes it works. More often it only gives you a useful list but everything not on the list are still in the gray area.
- Avoid redundant abilities. If an ability gives the creature a boost to a stat that is already very good, it probably doesn't need it. Also, if an ability is very similar to another ability the monster has, it's probably a good idea to drop one or the other.
- If you give the monster an ability that has a real-world counterpart, make sure you get the facts straight. For example, a pheromone ”triggers a social response in members of the same species.” Emphasis mine. Pheromones, by the way, were quite popular.
- Don't waste words on describing what an ability doesn't do. As a general rule, an ability only does what it says it does. For example, ”There is no HD limit for this sleep effect.” is largely unnecessary unless of course the ability functions as the sleep spell.”
- Save for a few exceptions, all constructs have a Construction section. If your construct doesn't have one, it's most probably seen as a mistake. Even if you explain why your construct doesn't have one ”...the secret of making them was lost..” it may still be seen as a mistake.
- When you're designing special abilities, it pays to remember the ”unwritten rules” I've mentioned a number of times in my earlier blog posts. I can't stress it enough how important it is to always check out a few other creatures of the same type (and subtype) and analyze them. For example, if you're designing a fey, look up a few other fey creatures in the PRD. If you notice that all (or most) creatures of a certain type and subtype have a particular type of special ability, your monster should probably have it, too. If it doesn't, there should be a good reason for it.
- Abilities that transform a creature into something else were popular in Here Be Monsters. I like the visuals of a creature turning into a swarm or a mass of tentacles, but the presentation wasn't very elegant in most of the entries. The monster needs a second set of stats for the other form, which easily makes the stat block look cluttered with "+XX in swarm form" here and there. A more elegant solution is to put all the changed stats in one place as a mini stat block.
- Don't include too many effects or outcomes in an ability, for example ”it can do this and then this other thing happens, and if x is true, then do this, …” If an ability has a lot of different effects, divide it into separate abilities (or at least sub-abilities).
- Don't bury the mechanics of an ability in too much flavor text. It's
ok to include a little bit of flavor text,
especially if it helps the GM to visualize what the ability does.
- I don't recommend including abilities that require more than two rounds to use. Combat is usually over in four rounds, so it's very likely the monster will never get a chance to use the ability.
- For more information on good and bad ideas in special ability design, please also read the top 5 entries and the judges' comments. I haven't repeated my comments on those entries' special abilities here because reading them with the actual monster entries puts them into context and is therefore much more useful. It's also a good idea to browse Paizo's RPG Superstar forums and check out the judges' comments, here's a link to RPGSS 2014 monster round. http://paizo.com/rpgsuperstar/round2