perl double loop is very very slow

delacruz · 2008-07-05 20:40:43

i made this double loop in perl but it is very slow

#!/usr/bin/perl
#
use Math::Random;

open(G,">>outfile.txt"); # open my file named outfile.txt but in perl it has the name G
for $i(1 ..100000){           #rows
    for $j (0 .. 999) {    #columns
    @array[$j]=random_normal();
    
    }
print G "@array \n";
}

how can i impove this.

josomebody · 2008-07-05 22:48:33

A loop iterating 10^9 times is gonna be slow, and perl isn't exactly the quickest language. You could do it just as easily in C unless there's some reason you have to use perl (like if this is homework). Might help the speed a little. Other than that, it's about as optimised as it's getting as far as I can tell.

buttons · 2008-07-05 22:49:46

Write a C extension to do it. Interpreted languages have trouble with "inner loops" like this.

gnud · 2008-07-05 23:48:33

Yeah, I have to agree. I had some ideas about optimalization, but on my (ancient) machine, it was still only half done after almost 4 minutes. A straigth forward C implementation ran in 40 seconds

Daenyth · 2008-07-06 00:53:49

Write it in assembly. C is so slow.

delacruz · 2008-07-06 01:39:32

buttons wrote:

Write a C extension to do it. Interpreted languages have trouble with "inner loops" like this.

im new to all this programming. Do you mean to write the entire program in C?

I am able to do this and iterate the entire matrix fairly quick with matlab here is the code i use with matalb, it does all this in around 3.38 seconds, YES 3.38 secs

row=100000;       % rows
col=1000;     % columns
drift=.15;       % drift for my stock market simulation
vol=.2 ;              % volitility for my stock market simulation
delta_t=1/12;         % simulation is for one month
S = zeros(row,col+1);   
S(:,1)=25;   % gives the frist column a value of 25,  the ":" means for all rows
for j=2:(col + 1)
   S(:,j) = S(row,1) + S(row,1)* drift *(1/col+1) * delta_t + S(row,1) * sqrt( vol *(1/col)*delta_t) * randn(row,1);
end; %

i am just experimenting and i am trying to find another way of doing a simple monte carlo simulation. Honestly i need a program that i can do alot of math with. Is matlab the best to use for my needs?

EDIT: i think my above code is all wrong

row=100000; 
col=1000;
randn(row,col + 1);

now this matrix of 100000x1000 runs in 2.4 secounds

Last edited by delacruz (2008-07-06 01:54:27)

buttons · 2008-07-06 02:02:55

delacruz wrote:

im new to all this programming. Do you mean to write the entire program in C?

It's never necessary to write everything in C, unless you really want to.

If you're doing heuristic simulation to minimise some function, like in monte carlo or simulated annealing, you only need to make the cost function in C. The cost function is going to be where your program spends all its time, anyway.

For instance, I just wrote a simulated annealing algorithm to optimise the ordering of a phylogenetic tree. Here's my cost function in python:

energy  = lambda x, matrix: sum([ matrix[i][i+1] for i in range(len(x) - 1) ])**2

Which simply sums each successive pair of tree leaves given some tree order x. Over the course of the simulation, the program might need to execute this function a million, perhaps 1.5 million times before it's done. This takes ages.

I wrote JUST the cost function in C, and simulated annealing is literally 7 times faster. The relevant C code parts:

for( i = 0; i < (array1_size - 1); i++ ) {
        /* data is a pointer to the start of the array data
         * strides is the address offset between successive data elements in a contiguous array at dimension [i] */
        
        sample1_idx = *(int *)(python_array1->data + i*python_array1->strides[0]);
        sample2_idx = *(int *)(python_array1->data + (i+1)*python_array1->strides[0]);

        result += *(double *)(python_array2->data + sample1_idx*python_array2->strides[0] + sample2_idx*python_array2->strides[1]);
    }
return PyFloat_FromDouble(pow(result, 2));

I realise python isn't perl but both have very flexible C extension integration capabilities. Hell, you can even inline it. See here: http://www.perl.com/pub/a/2001/02/inline.html

Arch Linux

#1 2008-07-05 20:40:43

perl double loop is very very slow

#2 2008-07-05 22:48:33

Re: perl double loop is very very slow

#3 2008-07-05 22:49:46

Re: perl double loop is very very slow

#4 2008-07-05 23:48:33

Re: perl double loop is very very slow

#5 2008-07-06 00:53:49

Re: perl double loop is very very slow

#6 2008-07-06 01:39:32

Re: perl double loop is very very slow

#7 2008-07-06 02:02:55

Re: perl double loop is very very slow

Board footer