sunfire 480R and linux

This is a quick blog post in order to set things straight. Though all major results from google pointing to the contrary (that linux is uninstallable on a sparc machine) these days I came to a fully functional 480 server while using gentoo sparc autobuilds. Other linux distros (debian,redhat,centos,suse) could not even boot the machine :S

So gentoo is your friend.

If for any reason anyone is intrested on the specifics of the “operation” please write a follow up :D

The funniest configure message I’ve ever read :)


...
configure: Using system-installed FFMpeg code
configure: WARNING:
======================================================================
WARNING: you have chosen to build gst-ffmpeg against a random
external version of ffmpeg instead of building it against the tested
internal ffmpeg snapshot that is included with gst-ffmpeg.


This is a very bad idea. So bad in fact that words cannot express
just how bad it is. Suffice to say that it is BAD.


The GStreamer developers cannot and will not support a gst-ffmpeg
built this way. Any bug reports that indicate there is an external
version of ffmpeg involved will be closed immediately without further
investigation.

The reason such a setup can't be supported is that the ffmpeg API
and ABI is in constant flux, yet there aren't any official releases
of the ffmpeg library to develop against. This makes it impossible
to guarantee that gst-ffmpeg will work reliably, or even compile,
with a randomly picked version ffmpeg. Even if gst-ffmpeg compiles
and superficially appears to work fine against your chosen external
ffmpeg version, that might just not be the case on other systems, or
even the same system at a later time, or when using decoders,
encoders, demuxers or muxers that have not been tested.

Please do not create or distribute binary packages of gst-ffmpeg
that link against an external ffmpeg. Thank you!
======================================================================


checking for sed... /bin/sed
...

LOL that’s what I call programmers with humor :twisted: And of course build succeeds and ffmpeg packages works alright :D

(gentoo amd) cpuinfo linux

These days I am violating my shiny new pc in many ways (huge matlab datasets processing, with multiple concurrent vms running at 100%) so I would like to check out (without rebooting and entering the bios) some shit about my machine. Since I am totally F/OSS as far as the OS is concerned I don’t have the luxury of windows apps like cpuid, or whatever. So in order to check out about the o/c I had to find another way. And there is another way, two to be exact.

  • The first is the lshw [1] progie (you know the drill gentoo fans ;-) ) emerge -vuDtN sys-apps/lshw

It is a good idea to build the frontend also (via the gtk USE flag). lshw is great. it provides you with tons of information, the cli’s ouput may passed as input (piped?) for anykind of shit you have in mind in anykind of enviroment, and is open source code. lshw (also known as Hardware Lister) is a small tool to provide detailed information on the hardware configuration of the machine. It can report exact memory configuration, firmware version, mainboard configuration, CPU version and speed, cache configuration, bus speed, etc. on DMI-capable x86 and amd64 (tested on both works flawlessly). For instance,

phenom2 ~ # lshw -class cpu
*-cpu
description: CPU
product: AMD Phenom(tm) II X4 940 Processor
vendor: Advanced Micro Devices [AMD]
physical id: 4
bus info: cpu@0
version: AMD Phenom(tm) II X4 940 Processor
serial: To Be Filled By O.E.M.
slot: CPU 1
size: 3400MHz
capacity: 3400MHz
width: 64 bits
clock: 200MHz
capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp x86-64 3dnowext 3dnow constant_tsc rep_good nopl pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt

  • The second one is  a tool [2- 64bits]  [3 - 32bits] [4 - link to the amd’s search page] build by amd and for amd compatible cpus (haven’t tested it in intel archs though it would be funny) but unfortunatelly amd (what a surprise!) doesn’t provide exactly what one may want.

What I mean by that? If you download one of these distributions you  will find out that these are rpms [for those stuck with ebuilds and gentoo you can read more about rpm there 5] for two specific distros; Suse and Redhat! No gentoo, no slack no nothing… Not even a static build… Who the hell is the software release engineerer in amd? Of course there is a bypass to the rpm shit .

  • One may extract the files out of the stupid rpms easily though;
    1. rpm2cpio CPUInfo-2_2_0_42-SUSE10364bit-Public.bin.rpm | cpio -idmv
    2. if you don’t have cpio (for gentoo users that’s impossible since its in the system) install it and proceed as advised
  • and of course the extracted files are dynamically linked (YOU M#*$#@)DUMB*#$#*$)@# PLEASE PROVIDE US STATICALLY LINKED BINARIES OR GIVE OUT THE DAMNED SOURCE,s o it may or may not work in your platform depending on the libraries you have installed… sigh… (For me every single one worked)
    • CPUInfo is the "fantastic” executable the amd provides
      phenom2 dumb-developers # ldd CPUInfo
      linux-vdso.so.1 => (0×00007fffd35fe000)
      …[many libraries including kdemultimedia!!!]…
      libuuid.so.1 => /lib/libuuid.so.1 (0×00007f0dc65a9000)

Instead of a conclusion;  Linux users do yourself a favour and use truly opensource software. Not some binary shitty thing that a 13yr old kiddo written… And if you haven’t got the clue, USE lshw.

PS. If you dl’ed the amd’s rpm’s then you should see something similar to :

phenom2 amd # ls -l
σύνολο 6470
-rw-r--r-- 1 root root 611293 2008-11-18 10:52 CPUInfo-2_2_0_42-RedHatEnterpriseServer4U464bit-Public.bin.rpm
-rw-r--r-- 1 root root 1075290 2008-11-18 10:52 CPUInfo-2_2_0_42-RedHatEnterpriseServer564bit-Public.bin.rpm
-rw-r--r-- 1 root root 656880 2008-11-18 10:52 CPUInfo-2_2_0_42-SUSE102OSS64bit-Public.bin.rpm
-rw-r--r-- 1 root root 671294 2008-11-18 10:52 CPUInfo-2_2_0_42-SUSE10364bit-Public.bin.rpm

gentoo: gcc (auto) vectorization

First of all, for those wondering, [1]: vectorization is a process where a sequential program using a pair of operands on the same instruction in an iteration is transformed into a vector program where a single instruction can perform multiple operations or a pair of vector. In this notation by vector I call datasets of the same type.

In quite modern machines/architectures (namely x86) there exists such a vector processing(one or more ;-) ) unit utilized by the SIMD instructions set (usually SSE,SSE, see below). SIMD [2] is one of various ways implementing parallelism in computer hardware.  In this schema,some large set of CPU components perform the same task (operation) at the same time, each with different data leading to a high utilization of these components without wasting CPU cycles in the same operation.

For instance the following code is vectorized:

#define SIZE 1024
int a[SIZE], b[SIZE], c[SIZE], d[SIZE];

void linear ()
{
int i;

for (i=0; i<SIZE; i++)
{
a[i] = b[i] * c[i] + d[i];
}
}
If the parallelism is viable, then the compiler after the autovectorization process may produce  high performance code that makes high utilization of processor vector units. This certaintly is of supreme importance (to reduce the total instructions executed where applicable) in a high performance distribution. So after some reasearch,  I decided to enable on all of my gentoo machines the vectorization on the CFLAGS (the vectorization as one may see in its homepage is automatically enabled on the -O3 optimizations [3] ) . Gentoo should be the fastest linux distro around; END OF STORY.

CFLAGS="-march=nocona -mtune=nocona -O2 -pipe -fomit-frame-pointer -ftree-vectorizer-verbose=5 -ftree-vectorize -fassociative-math -msse -msse2 -msse3"
The options to do so is the -ftree-vectorize combined with tha -msse -msse2 options (which are the SIMD utilizers for the x86 (amd64) arch) if you are having a more capable SIMD instruction set you may use it (i.e. -msse3, -msse4.1, -m3dnow(:-P) etc), while -fassociative-math can be used instead of -ffast-math to enable vectorization of reductions of floats.

here are some before and after results concerning vectorization speedup  on:

rodos ~ # uname -a
Linux rodos 2.6.27-gentoo-r8-korki-pentium-prescott #8 SMP Mon Mar 9 19:28:11 EET 2009 x86_64 Intel(R) Pentium(R) 4 CPU 3.00GHz GenuineIntel GNU/Linux

The results follow in the following snippet and are for 8192 bytes in kilo

Vectorize Enabled Vectorize Disabled
md2 7135,23 6435,79 0,11
mdc2 9210,54 9404,42 -0,02
md4 901018,97 636548,84 0,42
md5 546226,18 306845,91 0,78
hmac(md5) 551026,69 551318,87 0
sha1 299592,36 296080,73 0,01
rmd160 174093,65 148373,5 0,17
rc4 439552,68 439959,55 0
blowfish 99672,06 98432,34 0,01
aes-128 111878,14 111209,13 0,01
aes-192 102440,96 100368,38 0,02
aes-256 91635,71 91280,73 0

or in a more comprehensible representation:

openssl performance

gentoo howto: download and transcode any youtube video…

Hi there all,

in this article we will cover how to  download and transcode into any format{avi,mp3,…} in a few commands :-) This article is divided into two pieces; The init phase where all the necessary tools are build, and then some basic explanation of the tools usage.

We begin this article with the usual update your portage tree with
#emerge --sync
next you have to download the youtube-dl package. While it is possible to create a similar script with more or less functionality (i.e. using python and browser @ mechanize package) the point here is to be as simple as possible. So we download the package with
#emerge youtube-dl
Then we have to download some packages to manipulate the flv videos. Such tools (but not limited to) are ffmpeg and transcode. We download each one with the appropriate emerge command; Keep in mind that we have to setup the build flags on each one in order to have the ability to manipulate more than one datafile. In my system the /etc/portage/package.use contains the following on ffmpeg and transcode;
media-video/transcode X iconv jpeg mmx sse sse2 truetype 3dnow a52 altivec dv dvdread extrafilters fame imagemagick lzo mjpeg mp3 mpeg network ogg oss quicktime sdl theora\
v4l2 vorbis xml xvid

media-video/ffmpeg a52 aac amr encode ieee1394 imlib mmx network ogg oss sdl theora threads truetype v4l vorbis x264 xvid zlib
after setting up the /etc/portage/package.use for our tools then we emerge our packages with;

#emerge ffmpeg
#emerge transcode

This concludes our init/setup phase. And now we enter the tools usage part.
Scenario a.
Since we have forgot our Load CD and we desperately want to listen to until it sleeps track. No stress dude! This is a 1 minute task!

  1. We locate the video in the youtube (eg http://www.youtube.com/watch?v=sZtszXnG11E ) then we call the youtube-dl with argument this url; $youtube-dl -b -t http://www.youtube.com/watch?v=sZtszXnG11E (-b is for the best quality -t is for the title, since we only wanted to make an mp3 from this track we could omit -b argument with no significant loss in the quality. -t is to create a filename which contains part of the video title). If we download with -b switch bear in mind that our media is saved as mp4 video file. at programs exit the youtube-dl reports the file that has been saved eg;

    Retrieving video data: 100.0% ( 17.21M of 17.21M) at 125.20k/s ETA 00:00 done.
    Video data saved to sZtszXnG11E.mp4

    the sZtszXnG11E.mp4 is the filename we are gonna tamper :-)
  2. We transcode the sZtszXnG11E.mp4 with ffmpeg with one simple command; $ffmpeg -i sZtszXnG11E.mp4 -f mp4 -vn -acodec copy UntilItSleeps.mp3
    1. If we had downloaded the same file without -b switch the input file (-i sZtszXnG11E.mp4  in the previous bullet) would be something like sZtszXnG11E.flv. In that case we change the input codec parameter from mp4 to mp3 ; $ffmpeg -i sZtszXnG11E.flv -f mp3 -vn -acodec copy UntilItSleeps.mp3. We do so because the audio data are represented differently in flv (mp3) and mp4 (mp4) files respectively.

If anyone has any other scenario in mind please ask :-)

/etc/fstab mount options

Yesterday I had to rearrange my data over my hard disks and since zfs [2][3] is not yet 100% efficient in linux distros[4] I have to do all the dirty work manually…

My first question is how the hell does the stupid fstab accept the bind option. Since its man page says as much as nothing [5]

The quick and dirty way into finding out how the stupid fstab syntax works is to mount a filesystem normally as you would do via the mount command (bind,loop anything!!!) and then check the /etc/mtab file to understand the syntax! its freaking cool and easy then to copy paste the mtab directives into the fstab file and voila! no fstab related reading is necessary!

for instance or if you prefer it as an example after some mount deeds the system reports the following on the mtab


msi939 ~ # cat /etc/mtab
...
/dev/hdb1 /mnt/hdb1 reiserfs rw 0 0
/dev/sda1 /mnt/sda1 reiserfs rw,noatime 0 0
/dev/sdc1 /mnt/sdc1 ntfs rw,noatime 0 0
/dev/sdd1 /mnt/sdd1 ext3 rw,noatime 0 0
/mnt/hdb1/mirror /home/mirror none rw,bind 0 0

which in turns translates (after using UUIDs) to the following fstab:


UUID="064368f6-a696-44b3-8a84-31e47c16d7cb" none swap sw 0 0
UUID="cb0b34366-9f99-4394-8c0d-aaa9e3c7503c" /mnt/hdb1 reiserfs noatime 0 0
UUID="4aa2348-bc8a-437d-8d72-59c6c364e0b4" /mnt/sda1 reiserfs noatime 0 0
UUID="6AE34GFCE42BC8E1" /mnt/sdc1 ntfs rw,noatime 0 0
UUID="ac7asd3d-69a6-424c-bd3c-fb0ae35a813e" /mnt/sdd1 ext3 noatime 0 0
/mnt/hdb1/mirror /home/mirror none bind,defaults 0 0

(if by accident it happens of your UUIDs matching one of the above don’t worry, it ain’t a bug! I simple changed a bit the UUIDs with random keystrokes :P - If you do have a match, bear in mind that this is something really rare, so send me a mail, maybe we are meant to get acquainted :))

I hope you enjoy this fragment of knowledge, a simple thank you is sufficient :-)

gentoo and eclipse and pdt is mission impossible.

This is a quicky one, due to lack of time.

I tried for the past 24h to install eclipse + pdt on a couple of gentoo boxes I operate for home use. I could describe myself  at least as “experienced user“, and yet I couldn’t make it happen the right gentoo way [*]. After spending about 4-5 hours guessing what kind of dependencies should be sufficient to satisfy in order to eventually install the pdt, since the update manager of eclipse [3.2] provided by gentoo stable is a bit old and could not pick them automatically.

Well dudes if you are here in order to install eclipse and pdt on a gentoo system skip the stupid eclipse ebuilds (totally unmaintained maybe?) and go grab eclipse from the source: >>>>> READ HERE AND FOLLOW THE INSTRUCTIONS. IT DEFINATELY WORKS!!!! <<<<<.

Yeah I ebuilded the eclipse from the sources (20 mins - no ccache no distcc, it is java dude :( ) in order to have an (obsolete) cvs browser and a C dev platform…. Yeah right…

Conclusion gentoo + eclipse ebuild = bad idea… Do it the Clint Eastwood way: download unpack use…

*The right gentoo way implies the respect to ebuilds and follow the exact instuctions from the official sources/package maintainers etc…

gentoo xorg: Caught signal 11. Server aborting

One of the problems I had recently on a x86_64 gentoo installation with an ati 1550 graphics adapter was the constant Xorg-server crashing with the following message;

Fatal server error:
Caught signal 11. Server aborting

That message was haunting the damned Xserver and strangely enough, I had never seen it in my linux quest through the years. When the X couldn’t load it never segfaulted. During my courses at ceid and the C language classes [1][2] [and others], I had the strong belief that a segfault (signal 11) occurs only due to a PEBKAC. Namely from poor programming practice or from incorrect linking.
Read more

gentoo: unexpected slow down and other unexpected behaviour

It was sometime since I returned from vocation and I hadn’t use the gen64 system for a demanding task. Recently I had to do some simulations with ns2 and the performance was really poor! Meaning that some simple scenarios that demanded about 30 seconds (on the same system) required about 6-7minutes to complete. This utterly irritating behavior was most unexpected so actions should be taken to revert it.

Read more

Gentoo: speedup your compilations (1)

Hi all,

this is supposed to be some kind of tutorial series that tries to focus into the only disadvantage that gentoo has. The wasted cpu cycles due to constant compiling. And constant compiling means less productivity and of course more energy consumption. The last one is quite critical now days for home users or for operators owning a dedicated server in a datacenter with some power management rules.

The solutions for these so far that I have thought of are two (if you have something else on your mind please be my guest and comment in order to add it in the series)

  1. ccache
  2. distcc
  3. combination of the above

Read more