You are most likely trying to capture a game at a resolution higher than the video game supports. Roxio Game Capture only supports a resolution up to 1080i if the game does. However, most of modern HD games support 720 natively. You can generally see what resolution your game supports on the back of the game's packaging.
Here's how to check the TV output settings on your PS3:
- Make sure Standard Resolution (480i), 480, 720p and 1080i are checked. This will depend on the resolution of your TV supports. For example, if you have a 720 flat screen, you will not be able to capture in 1080i provided your game supports the resolution.
- Uncheck 1080p. If the original connection from PS3 to TV is through HDMI, it is likely you have this setting checked if you have a more recent HDTV.