Building rpms

Discuss software development issues here.

Building rpms

Postby aguilarojo » 09 Jun 2009, 01:08

Background of the problem:
Sometimes a package available within a YDL release is old in comparison to what may be current. For example to use the current release of GPG2 which allows for use of OpenPGP one of the libraries I need is libgcrpyt-1.4.4 but I need to have the version installed by anaconda removed. It just so happens that removing it using yum affects over 300+ other installed packages, which means that I have to find a means of having those same 300+ packages find and discover the new version of libgcrypt!

I tried researching this problem in other Linux forums and references and have learned much about spec files and so on. What I haven't found, even on IBM's website discussing rpm, is a logically procedural based answer.

I'm hoping that my query can be addressed here. Any takers?

Everything on the Earth has a purpose.
Every disease an herb to cure it.
And every person has a mission.
This is the Indian Theory of Existence.
-- Morning Dove, Salish (1888-1936)
User avatar
aguilarojo
ydl guru
ydl guru
 
Posts: 227
Joined: 06 May 2009, 14:50
Location: New York City

Re: Building rpms

Postby ppietro » 09 Jun 2009, 05:09

aguilarojo wrote:Background of the problem:
Sometimes a package available within a YDL release is old in comparison to what may be current. For example to use the current release of GPG2 which allows for use of OpenPGP one of the libraries I need is libgcrpyt-1.4.4 but I need to have the version installed by anaconda removed. It just so happens that removing it using yum affects over 300+ other installed packages, which means that I have to find a means of having those same 300+ packages find and discover the new version of libgcrypt!


Well - not exactly. All 300+ of those programs are using the only registered version of libgcrypt on your system - most likely libgcrypt-1.2.3-1. That's the way RPM works - when you install libgcrypt with RPM, it updates a database so that any other RPM based installers can find it.

Internally, RPM installs have a dependency requirement and a minimum version. So - all of those 300+ packages require libgcrypt - but you'd have to look at each package's original spec to see what minimum version of libgcrypt they require. When you go to uninstall something, it checks that RPM database, and prints warnings if any other RPM installs depend on it..

You can see the registered versions of all the files RPM knows about by using rpm -qa

I use this to find which version of something is installed by filtering it with grep. i.e.:
rpm -qa | grep libgcrypt

The nice thing about this is that you can update registered RPMs with newer versions with a simple download and RPM command.

As an example, if you had a PowerPC RPM of libgcrypt-1.4.4, you could just update the RPM database and all associated files with rpm -Uvh libgcrypt-1.4.4.ppc.rpm (or whatever the downloaded RPM is called)

Of course, if the new libgcrypt has other dependencies, it won't install unless all the other dependencies are resolved. This is why we use YUM - it automates all this downloading for us. :D

I tried researching this problem in other Linux forums and references and have learned much about spec files and so on. What I haven't found, even on IBM's website discussing rpm, is a logically procedural based answer.

I'm hoping that my query can be addressed here. Any takers?


The best place to find info about RPM is on Red Hat websites, not IBM. They invented it. (RPM originally stood for RedHat Package Manager - now it stands for RPM Package Manager. Recursive acronyms are fun! :D )

As for what you'd need to do - if you can't find a newer PowerPC RPM of libgcrypt to download - you'd need to build a new version from source. Then, you'd have a couple of choices, depending on how much source control you want on your system.

Option A - you configure your source code to build in an unused directory then edit /etc/ld.so.conf to point at it. This way, you can build newer libraries without disturbing your base YDL libraries.

An example of this is here:
viewtopic.php?f=19&t=3793&p=17013#p17013

You will have to build all dependent libraries with this method - but you won't trash your YDL install this way.

Option B - take the source code, build it, then create an RPM from it. This is harder than it sounds - well - for me anyway. billb has done this, I think. It's not for the faint of heart. :D

Option C - rebuild an SRPM from a newer distribution, then install it. For example - if you go here:
http://mirrors.versaweb.com/centos/5.3/os/SRPMS/
you can see that there is an SRPM for libgcrypt: libgcrypt-1.2.4-1.el5.src.rpm

Building SRPMS isn't too difficult - you create a series of subfolders on your hard disc in a folder. I use a folder in my home directory called rpm for this, then the subfolders - in all caps - are BUILD, RPMS, SOURCES, SPECS, SRPMS. After that, create a file called .rpmmacros in your home directory that tells RPM where to find your build tree. Mine looks like this:
Code: Select all
%_topdir /home/paulp/rpm


Then, download the SRPM wherever you like, and build it with this command:
setarch ppc32 rpmbuild --ppc --rebuild foo.src.rpm

Where foo.src.rpm is the name of the SRPM you downloaded.

We've found that you need the setarch ppc32 and --ppc arguments to correctly build for PowerPC.

After it finishes, you can just install it with rpm -Uvh newrpm.rpm

This would be the best way to do it. This is the way I generate the community builds of Firefox 3 for YDL directly from the CentOS repositories.

Cheers,
Paul
User avatar
ppietro
Site Admin
Site Admin
 
Posts: 4965
Joined: 13 Sep 2007, 22:18

Re: Building rpms

Postby aguilarojo » 09 Jun 2009, 17:10

ppietro wrote:
... All 300+ of those programs are using the only registered version of libgcrypt on your system - most likely libgcrypt-1.2.3-1. That's the way RPM works - when you install libgcrypt with RPM, it updates a database so that any other RPM based installers can find it.


Thanks for your effort and time for providing a very clear explanation! I really appreciate that you highlighted these and other details!

ppietro wrote:As an example, if you had a PowerPC RPM of libgcrypt-1.4.4, you could just update the RPM database and all associated files with rpm -Uvh libgcrypt-1.4.4.ppc.rpm (or whatever the downloaded RPM is called)

Of course, if the new libgcrypt has other dependencies, it won't install unless all the other dependencies are resolved. This is why we use YUM - it automates all this downloading for us. :D


I understand. However a PowerPC SRPM of libgcrypt-1.4.4 doesn't appear to exist anywhere, yet. I looked in Fedora but I haven't seen it there. Besides the current version of GPG2 requires nothing lower than libgcrypt 1.4. in it's build. So I'll have to explore option A or B, instead of C (which builds from an SRPM which already exists). So I would appreciate an elaboration of option B; if you left something out regarding option A I would deeply appreciate an elaboration there as well.

ppietro wrote:The best place to find info about RPM is on Red Hat websites, not IBM. They invented it. (RPM originally stood for RedHat Package Manager - now it stands for RPM Package Manager. Recursive acronyms are fun! :D )


Given that information on the web is so fluid; I appreciate that you provided a historical context to the development of this acronym.

I suppose I'm going to have to build an SRPMS from the source I'm acquiring from various original projects and their dependencies.

You got Foxfire 3? Great! As I get up to speed understanding which procedures to follow in building functional rpms from current source, I look forward to eventually posting my own SRPMS wherever you posted Foxfire 3 for the benefit of the PowerPC community. I appreciate that option A leaves the YDL installation undisturbed while permitting current sources and rpms to be built, recognized and installed without interfering with the formal YDL build. I'd prefer to proceed in the direction of allowing the newer builds to be used within YDL without disturbing the formal base installation.

Thanks again for your effort; I look forward to learning more.

Derick.

Everything on the Earth has a purpose.
Every disease an herb to cure it.
And every person has a mission.
This is the Indian Theory of Existence.
-- Morning Dove, Salish (1888-1936)
User avatar
aguilarojo
ydl guru
ydl guru
 
Posts: 227
Joined: 06 May 2009, 14:50
Location: New York City

Re: Building rpms

Postby ppietro » 09 Jun 2009, 19:01

aguilarojo wrote:You got Foxfire 3? Great! As I get up to speed understanding which procedures to follow in building functional rpms from current source, I look forward to eventually posting my own SRPMS wherever you posted Foxfire 3 for the benefit of the PowerPC community. I appreciate that option A leaves the YDL installation undisturbed while permitting current sources and rpms to be built, recognized and installed without interfering with the formal YDL build. I'd prefer to proceed in the direction of allowing the newer builds to be used within YDL without disturbing the formal base installation.


Hiya!

Yeah - YellowDog/Fixstars is actually fairly good about keeping Firefox up-to-date. However - we've noticed that their parent distribution - CentOS - updates Firefox a little more regularly.

So - I take the CentOS Firefox SRPMs and rebuild them for YDL. Building from CentOS avoids some of the library issues you get building from Fedora or other distributions who are expecting newer libraries. (Quick background - YDL was based on Fedora - now they're based on Red Hat Enterprise Linux via the CentOS recompile. RHEL runs about 3 versions or so older than Fedora: http://en.wikipedia.org/wiki/Red_hat_en ... tributions )

I donate my Firefox 3 builds to our own billb's ps3bodega repository. billb has been taking the old dribble and livna RPMs we need to work with YDL and consolidating them into one repo, updating packages where applicable. The instructions are here: http://pleasantfiction.ipower.com/bodeg ... ?f=28&t=21

I'm sure billb would be happy to add any new community generated RPMs. :D

Cheers,
Paul
User avatar
ppietro
Site Admin
Site Admin
 
Posts: 4965
Joined: 13 Sep 2007, 22:18

Re: Building rpms

Postby aguilarojo » 09 Jun 2009, 21:13

ppietro wrote:
I'm sure billb would be happy to add any new community generated RPMs. :D

Cheers,
Paul


Thanks Paul. Fascinating. Looks like thorough work. However I'm running YDL 6.1 on an Apple G4 1.67GHz Powerbook. I'm pretty clear that I could produce binaries and source which would work on the PS3. I'm not sure that software specifically designed for the PS3 however could run on my system. It depends if the binaries have been built to be processor specific implementing or calling the various SPUs; if the code has not been built that way then PowerPC binaries built on the PS3 should be able to run within my older G4. Likewise the source, if the source is not invoking Cell specific calls, I should be able to compile it, theoretically. It depends what's been done.

I guess we'll find out.

Derick.

Everything on the Earth has a purpose.
Every disease an herb to cure it.
And every person has a mission.
This is the Indian Theory of Existence.
-- Morning Dove, Salish (1888-1936)
User avatar
aguilarojo
ydl guru
ydl guru
 
Posts: 227
Joined: 06 May 2009, 14:50
Location: New York City

Re: Building rpms

Postby ppietro » 09 Jun 2009, 21:50

aguilarojo wrote:Thanks Paul. Fascinating. Looks like thorough work. However I'm running YDL 6.1 on an Apple G4 1.67GHz Powerbook. I'm pretty clear that I could produce binaries and source which would work on the PS3. I'm not sure that software specifically designed for the PS3 however could run on my system. It depends if the binaries have been built to be processor specific implementing or calling the various SPUs; if the code has not been built that way then PowerPC binaries built on the PS3 should be able to run within my older G4. Likewise the source, if the source is not invoking Cell specific calls, I should be able to compile it, theoretically. It depends what's been done.


You shouldn't have to do anything special for the PS3. Most of the YDL code we use is machine agnostic, and only runs on the PowerPC core of the Cell, which is fully Power ISA v2.0.3 compatible. Code you generate for YDL on your Apple should run just fine on our PS3s, assuming it doesn't use OpenGL. :D

There's little to none PS3 specific code in our standard repositories, either. The Dribble, Livna-stable & Fedora-extras repos we recommend installing are all generic, primarily Apple-based repos, not PS3 specific repos. They should work on your Apple just as well as they work on the PS3.

In fact, most of our repo code will work better on a PowerBook, since you have accelerated video. The PS3, due to hypervisor restrictions, only presents a simple frame buffer display. That means - no hardware OpenGL. :(

Linux on the PS3 relies heavily on the PowerPC core of the Cell. By using this core in conjunction with Sony's hypervisor layer, the PS3's admittedly odd architecture resembles a generic PowerPC computer - allowing fairly generic PowerPC Linuxes to run without modification. Unless code is written specifically to run on the SPEs on the Cell, they are unused. A standard program will not invoke them during compilation, since SPEs are not binary compatible with PowerPC op-codes, and require dedicated libraries.

So - yeah - your compilations should be fine for PS3s and Apples alike without any extra action on your part. :D

Cheers,
Paul

P.S. The reason I specify ppc32 build targets is not Apple or PS3 specific - it's PowerPC specific. Unlike x86-64, there are speed penalties running 64 bit PowerPC code. Unless you need the extra 64-bit memory space, or are dealing with 64 bit integer numbers - you're better off compiling PowerPC code 32 bit. :D
User avatar
ppietro
Site Admin
Site Admin
 
Posts: 4965
Joined: 13 Sep 2007, 22:18

Re: Building rpms

Postby aguilarojo » 10 Jun 2009, 02:25

ppietro wrote:You shouldn't have to do anything special for the PS3. Most of the YDL code we use is machine agnostic, and only runs on the PowerPC core of the Cell, which is fully Power ISA v2.0.3 compatible. Code you generate for YDL on your Apple should run just fine on our PS3s, assuming it doesn't use OpenGL. :D

There's little to none PS3 specific code in our standard repositories, either. The Dribble, Livna-stable & Fedora-extras repos we recommend installing are all generic, primarily Apple-based repos, not PS3 specific repos. They should work on your Apple just as well as they work on the PS3.

In fact, most of our repo code will work better on a PowerBook, since you have accelerated video. The PS3, due to hypervisor restrictions, only presents a simple frame buffer display. That means - no hardware OpenGL. :(


Thanks for clarifying PS3 specifications in comparison to G4. So they both have Altivec or the Velocity Engine as explained here:

http://en.wikipedia.org/wiki/AltiVec

I believe this also means that code developed using 128bit vector registers can be shared and utilized between our diverse systems, although because I have 1G of RAM which can be expanded up to 2 G, and the PS3 cannot -- any code written to take advantage of those registers would be faster on a G4 or PowerStation (which has two Altivecs see here: http://en.wikipedia.org/wiki/970MP#PowerPC_970MP) than it would be on a PS3 (because of the memory and hypervisor limitations). At least, there's a way out of those restrictions.

ppietro wrote:Linux on the PS3 relies heavily on the PowerPC core of the Cell. By using this core in conjunction with Sony's hypervisor layer, the PS3's admittedly odd architecture resembles a generic PowerPC computer - allowing fairly generic PowerPC Linuxes to run without modification. Unless code is written specifically to run on the SPEs on the Cell, they are unused. A standard program will not invoke them during compilation, since SPEs are not binary compatible with PowerPC op-codes, and require dedicated libraries.

So - yeah - your compilations should be fine for PS3s and Apples alike without any extra action on your part. :D

Cheers,
Paul

P.S. The reason I specify ppc32 build targets is not Apple or PS3 specific - it's PowerPC specific. Unlike x86-64, there are speed penalties running 64 bit PowerPC code. Unless you need the extra 64-bit memory space, or are dealing with 64 bit integer numbers - you're better off compiling PowerPC code 32 bit. :D


It's neat that although our systems our different the code we share will and can remain the same.
Last edited by aguilarojo on 10 Aug 2009, 02:50, edited 1 time in total.

Everything on the Earth has a purpose.
Every disease an herb to cure it.
And every person has a mission.
This is the Indian Theory of Existence.
-- Morning Dove, Salish (1888-1936)
User avatar
aguilarojo
ydl guru
ydl guru
 
Posts: 227
Joined: 06 May 2009, 14:50
Location: New York City

Re: Building rpms

Postby ppietro » 10 Jun 2009, 03:30

aguilarojo wrote:Thanks for clarifying PS3 specifications in comparison to G4. So they both have Altivec or the Velocity Engine as explained here:

http://en.wikipedia.org/wiki/AltiVec


Yes - see here:
http://en.wikipedia.org/wiki/Cell_(microprocessor)#Power_Processor_Element_.28PPE.29

Additionally, IBM has included an AltiVec unit which is fully pipelined for single precision floating point. (Altivec does not support double precision floating-point vectors.)


and
http://en.wikipedia.org/wiki/Cell_(microprocessor)#Overview

The PPE contains a 64-bit general purpose register set (GPR), a 64-bit floating point register set (FPR), and a 128-bit Altivec register set.


aguilarojo wrote:I believe this also means that code developed using 128bit vector registers can be shared and utilized between our diverse systems, although because I have 1G of RAM which can be expanded up to 2 G, and the PS3 cannot -- any code written to take advantage of those registers would be faster on a G4 or PowerStation (which has two Altivecs see here: http://en.wikipedia.org/wiki/970MP#PowerPC_970MP) than it would be on a PS3 (because of the memory and hypervisor limitations. At least, there's a way out of those restrictions.


Yes - that is my understanding as well. Any code using AltiVec should run on both systems, subject to the performance limitations you mention. It's when you get Cell specific code (i.e. code written specifically to run on the SPEs) that incompatibilities will occur, since G4s don't have SPEs. :D


aguilarojo wrote:It's neat that although our systems our different the code we share will and can remain the same.


Yup - it's pretty darned cool! :D

Cheers,
Paul
User avatar
ppietro
Site Admin
Site Admin
 
Posts: 4965
Joined: 13 Sep 2007, 22:18

Re: Building rpms

Postby aguilarojo » 10 Jun 2009, 04:14

Well, all the above makes pretty clear, to me anyway, that moving on building and sharing rpms is a really valuable endeavor especially if we can keep this at a high level of quality. While several ideas immediately pop into my head, I've got to get clear on how to best proceed regarding what we've covered in building rpms from source, etc because the latest SRPMS are not available from other distributions.

So doing things the hard way may be the only way for a time. If I can build up a familiarity with the process however, then I can build binaries or src.rpms which should be useful to everyone.

So I'll return with more questions later regarding the rpm packaging/development process.

I didn't think I'd get so excited by exploring what this entails but I'm glad we did.

Thanks again!

Everything on the Earth has a purpose.
Every disease an herb to cure it.
And every person has a mission.
This is the Indian Theory of Existence.
-- Morning Dove, Salish (1888-1936)
User avatar
aguilarojo
ydl guru
ydl guru
 
Posts: 227
Joined: 06 May 2009, 14:50
Location: New York City

Re: Building rpms

Postby billb » 24 Jul 2009, 12:11

aguilarojo wrote:I've got to get clear on how to best proceed regarding what we've covered in building rpms from source, etc because the latest SRPMS are not available from other distributions.


Even if there isn't an up to date SRPM available, an older SRPM is still useful in creating an updated package. The SRPM contains the source tarballs, any patches used, sometimes icons, and the .spec file which includes instructions for how to compile the application, where everything gets installed, how patches are applied, menu info, etc.

So, say you had an SRPM for foo 1.0, but you wanted to update it to foo 1.1. Assuming you already have your system set up for building RPMs (created directories, set location in your .rpmmacros file), you would install the SRPM:

rpm -ivh foo-1.0.src.rpm

This would install the source, any patches, the .spec file, etc. to the subfolders defined in your .rpmmacros file (in the %_topdir you've defined).

So you should have the source tarball (something like foo-1.0.tar.bz2) in your SOURCES folder, and the .spec file in your SPECS folder (foo.spec). In some cases all you'd need to do to create an updated package is copy the updated source tarball (foo-1.1.tar.bz2) to your SOURCES folder and update the version info in the .spec file, then build a new RPM from the .spec file like so:

setarch ppc32 rpmbuild -ba --ppc foo.spec

The -ba part tells rpmbuild to build binary (RPM) and source (SRPM) packages so if everything goes well you end up with foo-1.1.ppc.rpm and foo-1.1.src.rpm -- an updated RPM and SRPM package. The RPM will be in a subfolder in your RPMS folder, and the SRPM will be in your SRPMS folder.
PS3 60GB [CECHA01], FW 3.15, YDL 6.2, Samsung T260HD @ 1920x1200
Powermac G4 1.25 GHz x2, 2 GB RAM, YDL 6.2
User avatar
billb
Site Admin
Site Admin
 
Posts: 5522
Joined: 24 May 2007, 20:30
Location: Eastern NC, USA


Return to Software Development

Who is online

Users browsing this forum: No registered users and 3 guests