I have a similar setup:
Apple Macintosh G4/400
Apple Studio Display 17" CRT
ATI Rage 128 Pro
I tried installing YDL 4 a while ago. Had to use "install-safe" since both "install" and "install text" would present me with a blank screen. After a successful install the computer rebooted and I was presented with a login prompt. It flickered a few times as it tried to start the X server and then displayed a similar error message.
I logged in as root and changed the settings in my XConfig to 640x480 and 8-bit colour using "fbdev". The X Server stared but displayed only garbage. After a while I heard strange sounds from the built in speaker at which point I chose to press the reset-button on the computer.
I have previously tried YDL 3 which did slightly better. That is, I could start the X server using the above settings. Any changes however would result in an error. Neither the "r128" nor the "ati" driver would recognise the graphics card. Once in a while I've searched around with Google, but I've never found any solution that would let me run the X server at anything but 8-bit, 640x480.
So, sorry, no help from me...
Edit: Got X working. All it took was a reboot. So now I'm back in the same situation as with YDL 3.