Memory leak problem

Aletheuo · 2005-09-29 03:50:11

I'm trying to track down the source of a memory leak on my Arch box.
Is there some easy way to check memory usage of all processes periodically and write to a file or a program that does this already?
For a few weeks now, I've been noticing this memory leak, I have 1 gig of ram, I can reboot and leave my Arch box sitting at the login prompt, I can come back to it the next day and the memory usage is up in the 600-800meg range.
I've tried not starting all of my necessary daemons and shut everything down in the crontab and cron dir's, looking for errors in logs etc, everywhere I've looked everything seems normal to me.
My suspicion is it's the kernel or a kernel module. I keep my packages up to date so for everything thats installed I'm up-to-date, except my Nvidia package, it's still nvidia 1.0.7667-2.
The nvidia module is only using about 3.7meg and all other modules are under 100k of memory usage.
Any idea's what to check?

Thanks,
Craig

ghostwalker · 2005-09-29 04:00:44

You can try using 'top' to try and see which app continues to consume memory? Also do a lsmod to see what module are loaded? Netstat -tanup to see what protocols are loaded.

Aletheuo · 2005-09-29 14:14:03

Yea I've been using top periodically to check but I'm just not seeing the process that uses this memory up, thats why I think it's some kind of kernel leak possibly.

Craig

ghostwalker · 2005-09-29 15:58:21

What version of the kernel? What is you output of 'lspci -v' and 'lsmod'?

Aletheuo · 2005-09-29 16:17:41

Sorry for the long output but here's all you requested:

kernel: kernel26 2.6.12.2-1

lspci -v
00:00.0 Host bridge: Silicon Integrated Systems [SiS]: Unknown device 0746 (rev 10)
Subsystem: Silicon Integrated Systems [SiS]: Unknown device 0746
Flags: bus master, medium devsel, latency 32
Memory at e8000000 (32-bit, non-prefetchable) [size=64M]
Capabilities: [c0] AGP version 2.0

00:01.0 PCI bridge: Silicon Integrated Systems [SiS] SG86C202 (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 99
Bus: primary=00, secondary=01, subordinate=01, sec-latency=32
Memory behind bridge: ec000000-edffffff
Prefetchable memory behind bridge: e0000000-e7ffffff

00:02.0 ISA bridge: Silicon Integrated Systems [SiS]: Unknown device 0963 (rev 25)
Flags: bus master, medium devsel, latency 0

00:02.1 SMBus: Silicon Integrated Systems [SiS]: Unknown device 0016
Flags: medium devsel
I/O ports at 1400 [size=32]

00:02.5 IDE interface: Silicon Integrated Systems [SiS] 5513 [IDE] (prog-if 80 [Master])
Subsystem: Silicon Integrated Systems [SiS] SiS5513 EIDE Controller (A,B step)
Flags: bus master, medium devsel, latency 128
I/O ports at f000 [size=16]

00:02.7 Multimedia audio controller: Silicon Integrated Systems [SiS] SiS7012 PCI Audio Accelerator (rev a0)
Subsystem: Giga-byte Technology: Unknown device a002
Flags: bus master, medium devsel, latency 32, IRQ 18
I/O ports at d800 [size=256]
I/O ports at dc00 [size=128]
Capabilities: [48] Power Management version 2

00:03.0 USB Controller: Silicon Integrated Systems [SiS] 7001 (rev 0f) (prog-if 10 [OHCI])
Subsystem: Silicon Integrated Systems [SiS] 7001
Flags: bus master, medium devsel, latency 32, IRQ 20
Memory at ef003000 (32-bit, non-prefetchable) [size=4K]

00:03.1 USB Controller: Silicon Integrated Systems [SiS] 7001 (rev 0f) (prog-if 10 [OHCI])
Subsystem: Silicon Integrated Systems [SiS] 7001
Flags: bus master, medium devsel, latency 32, IRQ 21
Memory at ef000000 (32-bit, non-prefetchable) [size=4K]

00:03.3 USB Controller: Silicon Integrated Systems [SiS] SiS7002 USB 2.0 (prog-if 20 [EHCI])
Subsystem: Giga-byte Technology: Unknown device 5004
Flags: bus master, medium devsel, latency 32, IRQ 23
Memory at ef001000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [50] Power Management version 2

00:04.0 Ethernet controller: Silicon Integrated Systems [SiS] SiS900 10/100 Ethernet (rev 90)
Subsystem: Giga-byte Technology: Unknown device e000
Flags: bus master, medium devsel, latency 32, IRQ 19
I/O ports at e000 [size=256]
Memory at ef002000 (32-bit, non-prefetchable) [size=4K]
Expansion ROM at <unassigned> [disabled] [size=128K]
Capabilities: [40] Power Management version 2

00:0a.0 Modem: PCTel Inc: Unknown device 2189 (rev 04) (prog-if 00 [Generic])
Subsystem: PCTel Inc: Unknown device 1002
Flags: bus master, stepping, medium devsel, latency 32, IRQ 18
Memory at ef004000 (32-bit, non-prefetchable) [size=4K]
I/O ports at e400 [size=256]
Capabilities: [80] Power Management version 2

01:00.0 VGA compatible controller: nVidia Corporation NV11 [GeForce2 MX] (rev b2) (prog-if 00 [VGA])
Flags: bus master, 66Mhz, medium devsel, latency 32, IRQ 16
Memory at ec000000 (32-bit, non-prefetchable) [size=16M]
Memory at e0000000 (32-bit, prefetchable) [size=128M]
Expansion ROM at <unassigned> [disabled] [size=64K]
Capabilities: [60] Power Management version 2
Capabilities: [44] AGP version 2.0

lsmod
Module Size Used by
tun 12416 1
usbhid 48416 0
ehci_hcd 37256 0
uhci_hcd 33808 0
ohci_hcd 23044 0
snd_pcm_oss 55456 0
snd_mixer_oss 20480 1 snd_pcm_oss
snd_intel8x0 34752 0
snd_ac97_codec 83960 1 snd_intel8x0
snd_pcm 97796 3 snd_pcm_oss,snd_intel8x0,snd_ac97_codec
snd_timer 27012 1 snd_pcm
snd 60388 6 snd_pcm_oss,snd_mixer_oss,snd_intel8x0,snd_ac97_codec,snd_pcm,snd_timer
soundcore 10848 1 snd
snd_page_alloc 10244 2 snd_intel8x0,snd_pcm
nvidia 3713572 0
sis900 23552 0
mii 6272 1 sis900
usbcore 127740 5 usbhid,ehci_hcd,uhci_hcd,ohci_hcd

partial top output (showing the leaked memory usage):
top - 11:16:05 up 12:24, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 42 total, 2 running, 40 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0% us, 0.3% sy, 0.0% ni, 99.7% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 1033252k total, 746348k used, 286904k free, 373756k buffers
Swap: 1967952k total, 0k used, 1967952k free, 53352k cached

The Mem: used last night before I went to bed was ~91meg

phrakture · 2005-09-29 18:39:17

are you making sure you're actually using that much memory? make sure to subtract off buffers and cache, as those to count for resident size.

ghostwalker · 2005-09-29 19:30:08

Yeah, with the buffers and cache I really see no problems.

Here is my output of free:

total used free shared buffers cached
Mem: 515808 418480 97328 0 55740 271952
-/+ buffers/cache: 90788 425020 <---- the used is the total amount used a the given time
Swap: 1020116 1124 1018992

Aletheuo · 2005-09-29 21:58:58

I understand what you both are saying about the cache and buffer usage, but I still think this is a memory leak issue, this massive memory usage just started happening recently.
If I reboot this box, login and type top or free, I'll only see ~91meg memory usage, I don't think cache or buffers would make it jump to 746meg overnight with no activity on the box and nobody logged in/ssh'd in, the box just sitting there.

When I get back to the box in a couple of hours I'll copy&paste the top/free output of before I reboot the box and after, so you can see what it starts at and what it climbs to without any usage on the box.

phrakture · 2005-09-29 22:18:39

cgill27 wrote:

I understand what you both are saying about the cache and buffer usage, but I still think this is a memory leak issue, this massive memory usage just started happening recently.

Well, my "standard" memory usage including buffers/cache, is about 490/512. If you remove the buffers and cache the resident size is about 70ish.

cgill27 wrote:

If I reboot this box, login and type top or free, I'll only see ~91meg memory usage, I don't think cache or buffers would make it jump to 746meg overnight with no activity on the box and nobody logged in/ssh'd in, the box just sitting there.

That's totally normal. When the box is first booted it has nothing buffered or cached. Once it starts doing things, it keeps the those buffers filled, in case things are needed again, but they are not real memory usage.

To me this sounds like you've been using windows for too long. I remember when I first noticed the massive memory usage on my linux box. Then someone asked me a question which has become my favorite question: Ok, you see high memory usage... what problems is it causing?

Just because some piece of data or output says the word "error" or "warning" doesn't make it so. Are you experiencing any slow down? Are you experiencing any problems? Do things seem to segfault at random, or die with a message from the "oom-killer"?

Aletheuo · 2005-09-29 22:50:01

I even still think it's a memory leak, I've been using Linux for years, I know exactly what you speak of, thats why this is strange and I'm posting about it.
Keep in mind that this is behavior that changed recently, my memory usage was never really over 300-400meg with X running, Firefox, etc, I also try to keep my box lean and fast so thats why I noticed this.
I've got lots of Arch linux box's, in particular the one I'm on right now has 1.5gig ram with only 507meg used, and running on it is Zend Studio, Opera or Firefox, VMWare 5 with a WinXP virtual machine running with 384meg configured for it all right now as I type this with only 507meg used.
As you can see I fully utilize ram and I try to pay close attention to the box's stats as far as cpu/disk/ram.

I haven't experienced any show stopper with it yet because I reboot the box and use it normally and all the while the memory usage will be as low as the stats of the other box stated above.

I can use it for hours on end with low usage, it's only after I logout of X and leave the box overnight that the memory usage climbs.
I think I keep saying that.

Like I said if your willing I'll copy&paste stats for you to look at as I keep a journal of it's usage and you'll see just what I'm talking about.

phrakture · 2005-09-29 23:31:12

Hmm, it might be worth it to have a little log, including buffers/cache/total/used memory and maybe the top 3 apps or so... perhaps over the course of a day or so.... you could always have cron output free/ps output.

ghostwalker · 2005-09-30 00:51:52

Move upto the most recent version 2.6.14-rc2 and see if you get the same results?

sudman1 · 2005-09-30 10:41:10

cgill27, I have the same issues. I'm planning on doing a pacman -Syu to see if it will fix the issue.

Aletheuo · 2005-09-30 13:10:19

Last night I created a cron job to do a free/ps -ef every 30 minutes to a file, the cron task started about 11pm, between 2 and 2.5 hours later the memory usage jumped from ~89meg to ~739meg with the box just sitting at the console and no activity other than the normal daemons etc running in the background.

The files is 57kb but here is an excerpt from just before where the memory usage jumped up about 2.5 hours later. This is after a restart of the box.
The free/ps-ef script is /root/memchk.sh

total used free shared buffers cached
Mem: 1033252 89832 943420 0 5000 37092
-/+ buffers/cache: 47740 985512
Swap: 1967952 0 1967952
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Sep29 ? 00:00:00 init [3]
root 2 1 0 Sep29 ? 00:00:00 [migration/0]
root 3 1 0 Sep29 ? 00:00:00 [ksoftirqd/0]
root 4 1 0 Sep29 ? 00:00:00 [events/0]
root 5 1 0 Sep29 ? 00:00:00 [khelper]
root 6 1 0 Sep29 ? 00:00:00 [kthread]
root 8 6 0 Sep29 ? 00:00:00 [kacpid]
root 87 6 0 Sep29 ? 00:00:00 [kblockd/0]
root 136 6 0 Sep29 ? 00:00:00 [pdflush]
root 137 6 0 Sep29 ? 00:00:00 [pdflush]
root 139 6 0 Sep29 ? 00:00:00 [aio/0]
root 138 1 0 Sep29 ? 00:00:00 [kswapd0]
root 141 1 0 Sep29 ? 00:00:00 [jfsIO]
root 142 1 0 Sep29 ? 00:00:00 [jfsCommit]
root 143 1 0 Sep29 ? 00:00:00 [jfsSync]
root 144 6 0 Sep29 ? 00:00:00 [xfslogd/0]
root 145 6 0 Sep29 ? 00:00:00 [xfsdatad/0]
root 146 1 0 Sep29 ? 00:00:00 [xfsbufd]
root 746 1 0 Sep29 ? 00:00:00 [kseriod]
root 891 6 0 Sep29 ? 00:00:00 [ata/0]
root 908 6 0 Sep29 ? 00:00:00 [kcryptd/0]
root 909 6 0 Sep29 ? 00:00:00 [kmirrord/0]
root 917 6 0 Sep29 ? 00:00:00 [reiserfs/0]
root 970 1 0 Sep29 ? 00:00:00 udevd
root 2225 1 0 Sep29 ? 00:00:00 [khubd]
root 2905 1 0 Sep29 ? 00:00:00 /usr/sbin/syslog-ng
root 3107 1 0 Sep29 ? 00:00:00 /usr/sbin/smartd
root 3149 1 0 Sep29 ? 00:00:00 /usr/sbin/crond
root 3167 1 0 Sep29 ? 00:00:00 /usr/sbin/sshd
root 3199 1 0 Sep29 ? 00:00:00 /usr/sbin/httpd -k start
root 3216 1 0 Sep29 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe
nobody 3234 3199 0 Sep29 ? 00:00:00 /usr/sbin/httpd -k start
nobody 3235 3199 0 Sep29 ? 00:00:00 /usr/sbin/httpd -k start
mysql 3243 3216 0 Sep29 ? 00:00:01 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-
file=/var/lib/mysql/my-linux.pid --skip-locking --port=3306 --socket=/tmp/mysql.sock
root 3263 1 0 Sep29 ? 00:00:00 /usr/bin/rsync --daemon
root 3321 1 0 Sep29 ? 00:00:00 openvpn --config my.conf
root 3330 1 0 Sep29 vc/1 00:00:00 /sbin/agetty 38400 vc/1 linux
root 3331 1 0 Sep29 vc/2 00:00:00 /sbin/agetty 38400 vc/2 linux
root 3332 1 0 Sep29 vc/3 00:00:00 /sbin/agetty 38400 vc/3 linux
root 3387 3149 0 00:00 ? 00:00:00 /bin/bash /root/memchk.sh
root 3389 3387 0 00:00 ? 00:00:00 ps -ef
total used free shared buffers cached
Mem: 1033252 739412 293840 0 373656 48080
-/+ buffers/cache: 317676 715576
Swap: 1967952 0 1967952
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Sep29 ? 00:00:00 init [3]
root 2 1 0 Sep29 ? 00:00:00 [migration/0]
root 3 1 0 Sep29 ? 00:00:00 [ksoftirqd/0]
root 4 1 0 Sep29 ? 00:00:00 [events/0]
root 5 1 0 Sep29 ? 00:00:00 [khelper]
root 6 1 0 Sep29 ? 00:00:00 [kthread]
root 8 6 0 Sep29 ? 00:00:00 [kacpid]
root 87 6 0 Sep29 ? 00:00:00 [kblockd/0]
root 136 6 0 Sep29 ? 00:00:00 [pdflush]
root 137 6 0 Sep29 ? 00:00:00 [pdflush]
root 139 6 0 Sep29 ? 00:00:00 [aio/0]
root 138 1 0 Sep29 ? 00:00:00 [kswapd0]
root 141 1 0 Sep29 ? 00:00:00 [jfsIO]
root 142 1 0 Sep29 ? 00:00:00 [jfsCommit]
root 143 1 0 Sep29 ? 00:00:00 [jfsSync]
root 144 6 0 Sep29 ? 00:00:00 [xfslogd/0]
root 145 6 0 Sep29 ? 00:00:00 [xfsdatad/0]
root 146 1 0 Sep29 ? 00:00:00 [xfsbufd]
root 746 1 0 Sep29 ? 00:00:00 [kseriod]
root 891 6 0 Sep29 ? 00:00:00 [ata/0]
root 908 6 0 Sep29 ? 00:00:00 [kcryptd/0]
root 909 6 0 Sep29 ? 00:00:00 [kmirrord/0]
root 917 6 0 Sep29 ? 00:00:00 [reiserfs/0]
root 970 1 0 Sep29 ? 00:00:00 udevd
root 2225 1 0 Sep29 ? 00:00:00 [khubd]
root 2905 1 0 Sep29 ? 00:00:00 /usr/sbin/syslog-ng
root 3107 1 0 Sep29 ? 00:00:00 /usr/sbin/smartd
root 3149 1 0 Sep29 ? 00:00:00 /usr/sbin/crond
root 3167 1 0 Sep29 ? 00:00:00 /usr/sbin/sshd
root 3199 1 0 Sep29 ? 00:00:00 /usr/sbin/httpd -k start
root 3216 1 0 Sep29 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe
nobody 3234 3199 0 Sep29 ? 00:00:00 /usr/sbin/httpd -k start
nobody 3235 3199 0 Sep29 ? 00:00:00 /usr/sbin/httpd -k start
mysql 3243 3216 0 Sep29 ? 00:00:01 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-
file=/var/lib/mysql/my-linux.pid --skip-locking --port=3306 --socket=/tmp/mysql.sock
root 3263 1 0 Sep29 ? 00:00:00 /usr/bin/rsync --daemon
root 3321 1 0 Sep29 ? 00:00:00 openvpn --config my.conf
root 3330 1 0 Sep29 vc/1 00:00:00 /sbin/agetty 38400 vc/1 linux
root 3331 1 0 Sep29 vc/2 00:00:00 /sbin/agetty 38400 vc/2 linux
root 3332 1 0 Sep29 vc/3 00:00:00 /sbin/agetty 38400 vc/3 linux
root 3567 3149 0 00:30 ? 00:00:00 /bin/bash /root/memchk.sh
root 3569 3567 0 00:30 ? 00:00:00 ps -ef
total used free shared buffers cached
Mem: 1033252 739412 293840 0 373656 48080
-/+ buffers/cache: 317676 715576
Swap: 1967952 0 1967952
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Sep29 ? 00:00:00 init [3]
root 2 1 0 Sep29 ? 00:00:00 [migration/0]
root 3 1 0 Sep29 ? 00:00:00 [ksoftirqd/0]
root 4 1 0 Sep29 ? 00:00:00 [events/0]
root 5 1 0 Sep29 ? 00:00:00 [khelper]
root 6 1 0 Sep29 ? 00:00:00 [kthread]
root 8 6 0 Sep29 ? 00:00:00 [kacpid]
root 87 6 0 Sep29 ? 00:00:00 [kblockd/0]
root 136 6 0 Sep29 ? 00:00:00 [pdflush]
root 137 6 0 Sep29 ? 00:00:00 [pdflush]
root 139 6 0 Sep29 ? 00:00:00 [aio/0]
root 138 1 0 Sep29 ? 00:00:00 [kswapd0]
root 141 1 0 Sep29 ? 00:00:00 [jfsIO]
root 142 1 0 Sep29 ? 00:00:00 [jfsCommit]
root 143 1 0 Sep29 ? 00:00:00 [jfsSync]
root 144 6 0 Sep29 ? 00:00:00 [xfslogd/0]
root 145 6 0 Sep29 ? 00:00:00 [xfsdatad/0]
root 146 1 0 Sep29 ? 00:00:00 [xfsbufd]
root 746 1 0 Sep29 ? 00:00:00 [kseriod]
root 891 6 0 Sep29 ? 00:00:00 [ata/0]
root 908 6 0 Sep29 ? 00:00:00 [kcryptd/0]
root 909 6 0 Sep29 ? 00:00:00 [kmirrord/0]
root 917 6 0 Sep29 ? 00:00:00 [reiserfs/0]
root 970 1 0 Sep29 ? 00:00:00 udevd
root 2225 1 0 Sep29 ? 00:00:00 [khubd]
root 2905 1 0 Sep29 ? 00:00:00 /usr/sbin/syslog-ng
root 3107 1 0 Sep29 ? 00:00:00 /usr/sbin/smartd
root 3149 1 0 Sep29 ? 00:00:00 /usr/sbin/crond
root 3167 1 0 Sep29 ? 00:00:00 /usr/sbin/sshd
root 3199 1 0 Sep29 ? 00:00:00 /usr/sbin/httpd -k start
root 3216 1 0 Sep29 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe
nobody 3234 3199 0 Sep29 ? 00:00:00 /usr/sbin/httpd -k start
nobody 3235 3199 0 Sep29 ? 00:00:00 /usr/sbin/httpd -k start
mysql 3243 3216 0 Sep29 ? 00:00:01 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-
file=/var/lib/mysql/my-linux.pid --skip-locking --port=3306 --socket=/tmp/mysql.sock
root 3263 1 0 Sep29 ? 00:00:00 /usr/bin/rsync --daemon
root 3321 1 0 Sep29 ? 00:00:00 openvpn --config my.conf
root 3330 1 0 Sep29 vc/1 00:00:00 /sbin/agetty 38400 vc/1 linux
root 3331 1 0 Sep29 vc/2 00:00:00 /sbin/agetty 38400 vc/2 linux
root 3332 1 0 Sep29 vc/3 00:00:00 /sbin/agetty 38400 vc/3 linux
root 3570 3149 0 01:00 ? 00:00:00 /bin/bash /root/memchk.sh
root 3572 3570 0 01:00 ? 00:00:00 ps -ef

ghostwalker · 2005-09-30 13:13:50

You do have mysql, apache, rsync and openvpn loaded. Any one of these may be the problem? Those are the only apps outside the normal system.

Aletheuo · 2005-09-30 13:26:34

Right I know, but I've tried starting the machine up with those daemons set to not start and it is the same result, I'll do that memchk.sh test tonight if you want to see the results of that with the daemons set to not start.

phrakture · 2005-09-30 15:17:11

ghostwalker wrote:

You do have mysql, apache, rsync and openvpn loaded. Any one of these may be the problem? Those are the only apps outside the normal system.

Also... I don't run apache personally, so it may be normal - do you know you have 3 instances of it running? 2 as "nobody" and 1 as root

Aletheuo · 2005-09-30 15:28:18

The 3 instances of http are ok, in a normal install there are 8 I think, I have it scaled back since I'm only doing personal web development on this box, as far as the user it runs as I'm not sure on.
I think it starts the 1st http process as root and "spawns" the rest as user nobody or something along those lines. I'll check into that.

Craig

ghostwalker · 2005-09-30 23:00:00

What is you output of /proc/meminfo at bootup and then at eight hours?

Aletheuo · 2005-10-01 15:29:03

This is getting strange now, I setup a script to capture free/ps -ef//proc/meminfo every 30 minutes, I restarted the box, so I started with ~90meg used, and all night from about 1:30am to right now 10:30am it's still only ~90meg used.
So it looks like it has something to do with something that executes on a weekday, I'll know more tonight/tomorrow I guess.
I'm still not sure what it could be, I've combed through all the scripts in cron and the cron dir's.

If you want I can email you or IM you the txt capture file.

Arch Linux

#1 2005-09-29 03:50:11

Memory leak problem

#2 2005-09-29 04:00:44

Re: Memory leak problem

#3 2005-09-29 14:14:03

Re: Memory leak problem

#4 2005-09-29 15:58:21

Re: Memory leak problem

#5 2005-09-29 16:17:41

Re: Memory leak problem

#6 2005-09-29 18:39:17

Re: Memory leak problem

#7 2005-09-29 19:30:08

Re: Memory leak problem

#8 2005-09-29 21:58:58

Re: Memory leak problem

#9 2005-09-29 22:18:39

Re: Memory leak problem

#10 2005-09-29 22:50:01

Re: Memory leak problem

#11 2005-09-29 23:31:12

Re: Memory leak problem

#12 2005-09-30 00:51:52

Re: Memory leak problem

#13 2005-09-30 10:41:10

Re: Memory leak problem

#14 2005-09-30 13:10:19

Re: Memory leak problem

#15 2005-09-30 13:13:50

Re: Memory leak problem

#16 2005-09-30 13:26:34

Re: Memory leak problem

#17 2005-09-30 15:17:11

Re: Memory leak problem

#18 2005-09-30 15:28:18

Re: Memory leak problem

#19 2005-09-30 23:00:00

Re: Memory leak problem

#20 2005-10-01 15:29:03

Re: Memory leak problem

Board footer