There is an answer to this and a safety mechanism. The first part of this answer will address most cases, then the second part will address the edge cases.
For most things, you want to deliver a valuable feature with a backlog item. For example, you may have one that reads:
As a player, I want to be able to do a heavy attack and a light attack
so I can master more complex combat strategies than I could with
a single attack.
That involves all three roles. I assume our designer is going to help design the gameplay to be balanced and fun. The artist needs to create the animations and the programmer needs to implement it. Let's say this is a 5 for all of them to do all of the work to create this feature in the game.
Now let's take this one:
As a player, I want my equipment to visually change so I can see what I've
got equipped at any given time.
Now, as it is, this is open-ended because you have an almost infinite number of equipment pieces, so let's say to start for this backlog item, we're just going to do it with 3 weapons and 3 outfits. Now, for sizing, we can ask the team if this is a the same size (5), a little bigger (8), a lot bigger (13), etc. or even a little smaller (3).
This is how most sizing should go.
The Safety Valve
OK, so what if it just doesn't affect the other team members. Let's say we did that last one and the programmers made it that as long as you add the assets into the right folder, the game just pulls them in automatically, so this next story only has art work to be done:
As a player, I want a Halloween set of gear to celebrate the holiday
in a fun way.
Most planning poker decks have ? cards. If someone doesn't feel like they have anything to contribute, they hold up a ? card. This does two things. First, it doesn't muddy the estimation with wild guesses. Second, maybe the designer holds up a ? and the team says "Wait, you have work to do. These items need to be balanced." That's why I would go this way over simply asking the artist to estimate alone when you think it only involves them.
The Trap
One more trap to consider - don't assume people can't help because it isn't what they normally do. You can cross train. I expect that designer knows how to code games and if it's needed, they should step in and help. Not everyone can do art, but maybe they can help with color swaps or other simple tasks. Heck, old games did some of that stuff programatically. Especially when you have really diverse skills like you do in games development, it makes sense to respect the differing skillsets, but don't get trapped by them. Remember, there are a lot of one-man game developers out there. Most of the people on the team can learn most of the other skills if it helps the team to do so.