Paper Mario: The Origami King's battle system is certainly unique. Combining some of the old elements that made the franchise successful, as well as many new features.
What remains true is timed attacks that grant bonus damage. These well-timed attacks are called 'Excellent Attacks'.
They can be executed with any kind of attack, both using your boots to jump and your hammer.
Here's how to get excellent attacks.
How To Get Excellent Attacks In Paper Mario: The Origami King
To get Excellent attacks, you need to time your attacks. By this, you need to press the attack button, which is set to 'A', at the correct time.
When jumping, you need to press A just as you make contact with your enemy.
When swinging your hammer, you need to wait until the circles that form around it on the backswing turn red.
This will give you an excellent attack, if you're slightly off with the timing it will give you a great attack (which gives you a small damage boost) and if you miss the timing substantially you will just do base damage.
If you would like to practice this, you can do it at the Battle Lab in Toad Town. This is unlocked by saving the Toad, who has been turned into a dog, on Picnic Road.