OK the problem is you are using the cable that came with the PS3 which is a composite cable, the worst possible connection you can have! I think it's only 480i!
You do need to upgrade your connection, the component cables that work on the Playstation 2 will work on the PS3 so if you have a set of those around your good there but for the best picture quality get a HDMI cable
Like this one from Monoprice.com for less than $5, now if you don't have an HDMI receiver you will still need a audio connection, I use the Tos0Link (Fiber optic) cable) to get 5.1 sound.
Now your XBR will upconvert it to 1080p which is it's native resolution but it's upconverting a 480i signal so it will never look any better than a regular DVD without upgrading your connection.
Blu-Ray movies look amazing on my 70" XBR2 from my PS3 with HDMI.