Managing available memory (error 31)

[Loren Doornek]

I have a large program (406k) that recently started dying, very unceremoniously!  No error traps were triggered even though an error handler program and a SETERR were in place.  I added traces to the program, and found that it was dying at completely random places each time.  This leads me to believe that it was a memory issue (ie: ran out of memory). The program runs on Linux as a cron process, and the startup environment allocates 32m of memory to all of the processes.  I added a SET_PARAM 'SZ'=64000 into the routine to increase the available memory, and that seems to have resolved the problem.

However, I'd like to see if there is a better way to handle this.

The program is large since it encompasses several CALLed routines, and it will CALL itself several times using line labels.  (eg:  call PGN+";line_label",a$,b$,c).  I suspect that each time it calls itself, a new copy of the (large) program is loaded into memory, thus using up a lot more memory.  Some of the called routines also call other routines in the same program, so a call which is several levels deep could be loading several copies of the program into memory *if that is how the CALL command works*.

My question is: 
- When a program CALLs itself, does it load a new copy of the program into memory?
- And if so, will using ADDR to lock the program in memory prevent a new copy of the program from being loaded into memory?  (and thus reduce my need to allocate additional memory to the session)

Thanks in advance for any info you can provide!

[Mike King]

Loren,

Assuming you have program cache enabled and the program is NOT self modifying, the system will only ever load one copy of the program into memory regardless of how many times it called (recursively or not).  Obviously self modifying programs will result in multiple copies of the program being loaded.

As for using the ADDR directive, while you can load and lock a program into memory, we generally simply suggest enabling program cache (system parameter 'PC') as it handles all the housekeeping for you.

BTW: You mentioned your program is 406K, is that the file size or the program size?  Have you cleared the prior versions of the program from the file using the VER SINGLE command?  By default, program files automatically maintain prior saved versions of your programs in case you need to recall them.  While these prior versions do take up space in the program file, they are not loaded into memory.

[Loren Doornek]

Thanks for the info, Mike.  The program is not self-modifying.  We have versioning turned off, so there are no extra versions in the program. The 407k is the value I get from the PSZ variable.

Also, we have program caching turned off (I forget why we turned it off, but there was a reason long ago).  I added a line SET_PARAM 'PC'=10 to the program since it's a cron that will only apply that parameter to the single session, rather than all sessions. 

I did see that the process is still randomly dying, even with the 'SZ' parameter set to 64000, so maybe the memory isn't the issue.  I'll watch the process more to see if the 'PC' param makes any difference.

Thanks again, Mike.  You're wealth of knowledge is always appreciated!

[Stéphane Devouard]

Loren

You may want to try the IZ parameter as well, to have your program ignore any memory size limitations

Regards

[jasonc]

Loren,
You mentioned that that you added traces to your program and that it was dying at random places.

If you are using settrace or logging something to a file, I just wanted to confirm that you are turning off buffering on that file channel with the '-B' mnemonic.  Otherwise, it might appear to die at random places just because results were buffered and not written to disk before the process crashed.  Fair warning... it will slow down processing.

[Loren Doornek]

Stephane - I thought about using the 'IZ' switch, but I'd really rather figure out the problem before hitting it with that really big hammer of allowing unlimited memory.  I suspect that there is a memory leak somewhere (like using a "local" variable multiple times in a single routine, which I've seen in the past), and I'd rather find the problem if I can.  If not, I'll try the 'IZ' parameter to see if that helps.

Jason -  Thanks for that tip on buffering.  I don't have the -B switch turned on for the channel, so I'll try that to see if it gives me any different results.

[Loren Doornek]

Question regarding the -B mnemonic: 

Can that be applied to just the trace file by printing the mnemonic to the channel (ie: PRINT (tracechannel)'-B'), or does it apply to the entire session (ie: PRINT '-B')?

[jasonc]

A single channel...

open lock (5)"/tmp/crashlog"
print (5)'-B' ! Now anything sent to this channel will be written to disk immediately instead of buffered

[Loren Doornek]

Now this is getting interesting, since it doesn't seem to be a memory problem.

I followed all of the suggestions given:  Boosted the memory to 64megs, turned on the 'IZ' parameter to ignore memory limits, and set the '-B' flag on the trace channel.  The process is still dying at random points.  In my for most recent tests, the failing line of code was: trying to CLOSE a channel, PERFORMing a program/line_label, READing a channel to a data file, and setting some variables using an IOLIST.  The only thing in here that is remotely unique is the last item, which does LET KEY$=REC(IOL=KEY_IOL,SEP=$$), but that same code is executed multiple times in the program without a problem.

Since the 'IZ' parameter is set, it would seem to rule out this being a memory issue.

The program is large, and recently got larger.  Mike, we had discussed program size and variable name table size limitations in an earlier thread (https://forum1.pvxplus.com/index.php?topic=887.msg2983#msg2983).  I just checked the variable name table length using the info you provided in that thread, and the value returned is 29,015.

Is it possible that the variable name table size is causing this crash?  Any other suggestions on figuring out exactly what is causing this to crash?

[Mike King]

You could be hitting the limit for Variable names -- especially if new variables are added dynamically from data dictionaries or by code as in the VIA or EXECUTE directives.

[Loren Doornek]

Mike, there are no EXECUTE commands, but the program uses a LOT of files with data dictionary variables (ie: open the channel with IOL=*, then read the records, thus pulling in all of those variables).  The program also dimensions quite a few variables to the IOL of a channel.  (ie: DIM REC$:IOL(chan); READ RECORD(CHAN)REC$) 

Based on your comment, these types of variables are also using up space in the variable name table, correct?

If so, I'll probably need to split this program out into a couple of smaller programs to eliminate that problem.

[Loren Doornek]

Splitting the program to reduce the number of variables seems to have resolved the issue.  Thanks to everyone for all of your suggestions!

One additional question:  The program is importing a lot of JSON data, and some of the JSON files are fairly large.  I'm reading them into an array using DIM LOAD arr${all}=json$, and these arrays sometimes have 20,000+ elements. 

Are all of the individual elements of the array equally affecting the variable table?  Ie: does the variable table just consider "arr${all}" as 1 item, or does it track each element (arr$[1] ,arr$[2], arr$[3],...) separately?  Also, since these array elements have named fields, (eg: arr$["field.1.name"]), I'm assuming that these named fields also may affect the variable table.

I'm working on re-designing the program to avoid this problem, while also allowing room for future enhancements.  If the JSON arrays are having a huge effect on the variable tables, I may need to change the routines that request the JSON to request smaller chunks of data.

[Mike King]

Importing data from JSON using the DIM LOAD doesn't add variables.  It simply add associative keys to the variable array.  You can have an unlimited number of named element in an array.

(okay not unlimited -- up to whatever memory will hold)

[Loren Doornek]

Perfect!  Thanks, Mike!  I really wasn't looking forward to breaking up the JSON.  Kudos on the design of the associative arrays - they make handling the JSON a breeze, and the fact that they don't significantly affect the variable table is a relief.