3.5e dominate person and calm emotions—no interaction

In 3.5e, dominate person is not a charm spell, but rather a compulsion spell—the two have different rules and different effects. For that matter, calm emotions has nothing to do with either unless they specifically altered emotions—so calm emotions could suppress the rage spell,¹ also a compulsion, but does nothing against dominate person.

5e dominate person and calm emotions—total suppression

In 5e, dominate person applies the Charmed condition, and then also supplies an effect that reads “While the target is Charmed,” and calm emotions says it suppresses any effect that causes the target to be Charmed—which means that the “While the target is Charmed” clause is no longer met, so yes, 5e-calm emotions suppresses 5e-dominate person.

About being a “10th-level” spell—that sounds like a misreading

Dominate person isn’t a 10th-level spell in either edition—it’s 5th-level in both. However, in 3.5e, sorcerers get 5th-level spells at the 10th of that class—and thus the spell would have caster level 10th. That wouldn’t affect very much—just how close you have to be and how long it lasts. Meanwhile, 10th-level spells, as such, didn’t really even exist—there were rules for 10th-and-higher level spell slots, which meant you could cast lower-level spells from those slots for increased effect,² but it wasn’t the same as there being printed “10th-level spells.” There were also Epic spells, which counted as 10th-level spells, but used entirely different rules. So I kind of doubt that the module has dominate person as a true 10th-level spell, because that isn’t really a thing in 3.5e. It’s not impossible, but seems unlikely.

Whatever “10th-level” really means, it doesn’t protect against calm emotions

Whether we’re talking about caster level 10th or spell level 10th, that doesn’t really matter to 3.5e-calm emotions. For example, it does suppress all fear effects—even if they’re cast with caster level 10th, or Heightened to a 10th-level spell slot. Neither of those would matter, because calm emotions does not indicate any limitation that prevents it from suppressing fear effects based on level.

Conclusion: if using 5e-dominate person and 5e-calm emotions, should have total suppression

In 3.5e, calm emotions would not be a valid response to dominate person at all, but in 5e, it is. And in both editions, the spell level doesn’t matter at all for this purpose—calm emotions does not indicate any difficulty suppressing higher-level effects in either edition.

And I recommend using the 5e versions of both spells here, because you are playing with the 5e rules. In 3.5e, there were other, common effects that could block dominate person,³ but those aren’t available in 5e. You instead used the 5e spell that is supposed to protect against Charm effects, calm emotions. That was the correct choice, and it should have worked.

1. _{Probably; 3.5e is not always clear about what is and what isn’t an emotional effect. Calm emotions calls out the morale bonuses of the rage spell, but doesn’t mention the other effects.}




2. _{Using metamagic, which weren’t a sorcerer class feature but instead feats that made spells use higher-level spell slots in exchange for some increased effect. 3.5e also does not have default effects for casting spells from higher-level spell slots—even basic things like a higher saving throw DC, which was based on spell level, required the Heighten Spell metamagic feat.}




3. _{For example, the core 1st-level cleric spell protection from evil could block all domination—the 5e protection from evil and good only blocks the Charmed effects of certain creatures. And note the wording—like calm emotions, it blocks the Charmed status. This is the way 5e handles blocking domination, instead of having charm and compulsion as separate things.}

In 5e, yes

In 5th edition, Dominate Person is a spell which causes the charmed condition, and while it is charmed, you can give it commands:

You attempt to beguile a humanoid that you can see within range. It must succeed on a Wisdom saving throw or be charmed by you for the duration. [...]

While the target is charmed, you have a telepathic link with it as long as the two of you are on the same plane of existence. You can use this telepathic link to issue commands to the creature while you are conscious (no action required), which it does its best to obey.

If the target becomes not-charmed for any reason, the rest of the effects of the spell also stop working - they can only be ordered about while charmed.

Calm Emotions does indeed suppress any effect which causes the charmed condition, which includes Dominate Person, and so while the subject was under the effects of both spells, Calm Emotions would override Dominate Person:

You can suppress any effect causing a target to be charmed or frightened. When this spell ends, any suppressed effect resumes, provided that its duration has not expired in the meantime.

It doesn't matter how powerful the spell that produced the effect was or anything like that. Unless that particular effect itself says otherwise (which Dominate Person does not), it is suppressed by Calm Emotions.

In 3.5e, no

The 3.5e version of Dominate Person is a compulsion enchantment, but it is not a charm effect, and it is not subject to any protections against charms. Additionally, the 3.5e version of Calm Emotions does not suppress charm effects, but it would cause an affected subject to be unable to take aggressive actions against anyone, which is roughly how your DM eventually adjudicated the situation.

When combined, it's entirely up to your GM

When you mix and match the rules from very different editions like this, there are no hard and fast rules about how to adjudicate the interactions. Given that this is a case where both of the spells involved are present in updated form in the newer edition doing roughly the same thing, I would personally have stuck entirely to 5e's mechanics and allowed calm emotions to suppress the effect of dominate person.

In 5e, calm emotions suppresses dominate person at any level

Since you have said that you are trying to convert all rules to 5e and use all 5e versions of spells, I will answer from that edition exclusively.

Dominate person says:

It must succeed on a Wisdom saving throw or be charmed by you for the duration. [...] While the target is charmed [...]

And calm emotions says:

You can suppress any effect causing a target to be charmed or frightened.

Spell level does not matter. If it is a charm effect, calm emotions can suppress it.

Using 3.5 rules confuses the issue

You mention the dominate person spell being cast at 10th level, but that is not possible using 5e rules where 9th is the highest spell slot you can cast with.

So, depending how the DM is combining rules here, there is room for confusion based on 3.5's rules. Though, even in 3.5, this is not a 10th level spell so there may be even greater confusion happening.

Regardless, in 5e the answer is clear: calm emotions will suppress dominate person.

If you were obeying 5th Edition rules only, Calm Emotions should have worked

The rules for these two spells are pretty clear.

Dominate Person

You attempt to beguile a humanoid that you can see within range. It must succeed on a Wisdom saving throw or be charmed by you for the duration.

[...]

Calm Emotions

[...] If a creature fails its saving throw, choose one of the following two effects.

You can suppress any effect causing a target to be charmed or frightened. When this spell ends, any suppressed effect resumes, provided that its duration has not expired in the meantime.

[...]

Note however, that Calm Emotions does not 'dispel' the charm effect, it only suppresses it. So for the duration of Calm Emotions, the dominated person would be able to freely act, but if Calm Emotions were to end early through broken concentration, or if Dominate Person had been cast at a higher level (like your post suggests) causing its duration to be extended, then they would be Dominated again once Calm Emotions wore off.

Be wary about trying to mix features from different versions

3.5th edition does have the spell Dominate Person, as well as 5e. But while both allow control over the dominated target, they have different conditions affecting their use. In particular, the 3.5 version of Dominate person is far more difficult to remove, and generally a more powerful spell.

• 5e Dominate Person
  
  
  
  
  • Lasts 1 minute, with the duration extending based on the spell slot level being used
  
  
  • Targets repeat the saving throw any time they take damage
  
  
  • Applies the Charmed condition (lots of effects interact with the Charmed condition)
  
  
  
  


• 3.5e Dominate Person
  
  
  
  
  • Lasts 1 Day, duration extending based on the spell level it was prepared as
  
  
  • Targets repeat the saving throw "when they do something against their true nature"
  
  
  • Doesn't apply a generic condition
  
  
  
  

3.5 edition's Dominate Person is a lot more difficult to remove and counteract than 5th edition's is (at least in 5th edition), meaning if you introduce the 3.5 version into a 5e campaign, players are going to have a much more difficult time dealing with it. Since 5e does have an identically named spell with most the same effects, your DM should prefer using it, and also prefer 5e spells/items/effects wherever relevant, to make sure they aren't accidentally unbalancing their campaign modules.

Yes, assuming all 5th edition rules are being used

Given your question, this is a bit ambiguous.

Assuming 5th edition all the way with regards to rules and spell descriptions, then Calm Emotions would suppress Dominate Person.

Dominate Person states that the person is Charmed and the rest of the spell effects follow the statement:

While the target is charmed, ...

Whereas Calm Emotions specifically states:

You can suppress any effect causing a target to be charmed or frightened.

However, your description about using a mixture of rules does confuse the issue. The rules aren't generally going to work if you use the exact descriptions of rules, spell effects and so on from one version and try to relate that to descriptions from another version. Too much has changed - especially with things such as conditions (like Charmed) and how they apply to spells.

When translating, you have two choices - swap all the spells for their equivalent wherever possible. This is probably the easiest way to go, although spells may have gone up or down in level or changed so much that it makes some monsters easier or harder than originally intended.

The other option is just to use the old version of a spell 'as is' and accept the fact that some rules just won't apply and this situation will always just be a DM's call.