Speed up Java for Rapid Miner

[cleo]

Hello,

I have installed RapidMiner (www.rapid-com) an open source data mining software written in java on my ps3, and a different computer running ubuntu 10.04.

The same program running on the machine ubuntu takes about 30 mins, and over 2 days on the PS3. The program has very little graphics, and I thought that it should run faster on the PS3.

Do I need to compile the java code with any special libraries to take advantage of all the PS3 processors? Are there any suggestion to make this program run faster?

Thanks in advance,
Cleo

[ppietro]

Do I need to compile the java code with any special libraries to take advantage of all the PS3 processors? Are there any suggestion to make this program run faster?

There is no PS3-native Java virtual machine (VM), but we can use PowerPC VMs on the main core of the PS3's Cell - what we call the PPE. Unfortunately, we can't use Sun's Java, since they don't support PowerPC. However - we can use IBM's PowerPC Java - which is licensed directly from Sun.

By default, YDL ships with an extremely slow, reverse-engineered, open source PowerPC version of Java called GCJ. The first thing I would do is replace it with IBM's Java. We have instructions on how to install it here:

viewtopic.php?t=2935

NOTE: The instructions are for installing Java into your browser. It sounds like you may have installed this program - so you'll need to do one extra step.

Open a terminal window and type the following:

Code: Select all

su
ln -s /opt/ibm/java-ppc-60/jre/bin/java /usr/local/bin

The first command switches you to root user. You'll need to be root to do this. The next command links the IBM Java command to /usr/local/bin so that the IBM Java command runs before GCJ Java.

Other than making sure you're using IBM Java instead of GCJ Java, I'm not sure there's much else you can do. You may be running into memory limitation issues - the PS3 only has 256 Megs of system RAM. If your Java program uses more than this, you'll start swapping out memory to disk, which really slows it down.

BTW - currently, the only way to tap into the other Cell cores - what we call the SPEs - are to code directly for them in C/C++. No currently available Java uses them, although there are some non-released experimental Java VMs that do.

Cheers,
Paul

[cleo]

Thanks for the responce,

Do you think I could install cellvm to use all the processors?
http://people.inf.ethz.ch/anoll/publications/cellvm.pdf

http://forums.ps2dev.org/viewtopic.php?t=11188


Also when I run "Top" from Terminal, it tells me that most of my RAM is already used, so I think I am running into memory swap issues.

Would following these steps increase the RAM?
http://www.terrasoftsolutions.com/suppo ... -mem.shtml

Cheers,
Cleo

[ppietro]

Thanks for the responce,

Do you think I could install cellvm to use all the processors?
http://people.inf.ethz.ch/anoll/publications/cellvm.pdf

http://forums.ps2dev.org/viewtopic.php?t=11188

If you can find a copy - go ahead. Unfortunately, it was never really finished - it was more like a proof of concept, from what I understand. It certainly has not passed Sun Java certification like IBM's Java has.

BTW - how's the performance with IBM's PowerPC Java? Much faster than GCJ, right?

Also when I run "Top" from Terminal, it tells me that most of my RAM is already used, so I think I am running into memory swap issues.

Would following these steps increase the RAM?
http://www.terrasoftsolutions.com/suppo ... -mem.shtml

It would increase swap file RAM - but not main system RAM. Your PS3 can only address 256 Megs of system RAM - ever. It's hard soldered on the motherboard.

If you're running 6.2, you should already be using about 250 Megs of Video RAM as priority swap, with an additional 512 Megs of RAM on hard disc. This should be plenty for most programs - even Java applications.

Unfortunately, due to the size of Java's VM, you'll hit swap pretty quickly. The 250 Meg video RAM is the fastest swap you'll get on the PS3. As for the hard disc swap speed - performance can vary. Some hard discs are faster than flash memory and vice versa. If you feel comfortable trying the link above to set up Flash RAM as swap, try it - this is Linux after all - but I wouldn't expect much faster performance.

Cheers,
Paul

[cleo]

ello Paul,

Thanks again.

I installed IBM's PowerPC java, and linked it as above. I then run terminal and the command top. I then ran rapidminer from the a different terminal window, and it runs very quickly (noticable different with the default java VM) until it starts a memory swap. Then it crawls.

I was going to ask you why its reccomened to install the 32 bit version instead of the 64 bit until I saw the post viewtopic.php?f=4&t=3098&p=11887#p11892

I am new to linux, and was wondering if there is a distro for the PS3 that does not contain a GUI. I am hoping that would increase the available RAM to me. Also you said that there are some non-released experimantal java VMs that tap into the SPEs cores. Would you happen to know who I could contact about that?

Cheers,
Cleo

[ppietro]

I installed IBM's PowerPC java, and linked it as above. I then run terminal and the command top. I then ran rapidminer from the a different terminal window, and it runs very quickly (noticable different with the default java VM) until it starts a memory swap. Then it crawls.

Yup - that's about as good as you're gonna get on the PS3, I'm afraid.

I am new to linux, and was wondering if there is a distro for the PS3 that does not contain a GUI. I am hoping that would increase the available RAM to me.

You don't need a non-GUI Linux distro - you just need to turn off the GUI. All Linuxes are text based by default - we load the GUI as a later step.

A quick way to do this is to open a terminal window and type the following commands:

Code: Select all

su --login
init 3

Then, use your normal login at the text prompt.

To go back to GUI, use these commands:

Code: Select all

su --login
init 5

or just safely reboot with these commands:

Code: Select all

su --login
shutdown -r now

Also you said that there are some non-released experimantal java VMs that tap into the SPEs cores. Would you happen to know who I could contact about that?

You listed it above - I was referring to the CellVM.

Cheers,
Paul

[bonedome]

Hello
for a very small ram bonus read this viewtopic.php?p=14791#14791 and viewtopic.php?f=19&t=3497&start=45#p28041 gotta go dinner

[cleo]

Hello

Thank you bonedome,

I have stop the services listed in viewtopic.php?p=14791#14791 and have edited the files as described in viewtopic.php?f=19&t=3497&start=45#p28041.

After stopping the GUI as per Paul's instructions (which is much better then the idea I had on installing a different Linux distro) I ran the top command

Tasks 75 1 running 74 spleeping
Mem: 229176k total, 91476k used, 137700; free

I am running Yellow Dog 6.2 and using the IBM's Java. When I run rapidminer from command line the process takes about 4 hours and 55 minutes, and before these changes it took over 2 days. On my normal computer it takes 1 hour 15 minutes. Since I usually run several different processes at night the PS3 is now usufull to me Thank you both!

Any advice on killing any other processes or freeing up unused ram? I think rapidminer may have a memory leak as after the process is finished there is almost no free Mem.
Is it okay to kill python?

Also I am going to try to contact the authors of CellVM and see if I can get a copy from them.

Thanks for all your help,
Cleo