Backups and corrupted files

[Paul Robinson]

I have a client who is having problems with their system backup, specifically recoveries. The problem is when we attempt to recover our ProvideX Plus data files from the backup, selected data files restore as corrupted. These are files I am sure are active at time of backup. The client is unable to schedule a time in the day or night when they can get everyone off the server to do a clean backup. Theirs is a 24 hours operation. As a result, files are likely being modified during the backup process, and I suspect, corrupting the files. Has anyone experienced this and if so, come up with a way to address the problem? Their server is a CentOS Linux server and PvxPlus is 12.5.

[yonman]

How large are the corrupted files?  And are they consistently corrupted each time they are restored?

[Michael Greer]

Paul,
In my experience backing up a file that is actively being updated during the time it is actually being backed up will guarantee corruption.  I'm not sure of the exact sequence that PxPlus uses to update files (header locks, data writes, etc), but the scenario is simply this.  You have already backed up a part of a file which an update may change, and then will back up further toward the end of the file with "updated" data.  This clearly introduces internal inconsistencies which appear as corruption.

[Mike King]

Paul, have you considered using the CREATE FILE directive to make your backups?
https://manual.pvxplus.com/page/directives/create_file.htm

This directive (added in PxPlus 2016) was specifically designed to help with this problem.  It does a high-speed data backup of active data files while placing a temporary lock on the file that prevents any updates. 

Assuming your files are a reasonable size you should be able create duplicate files fairly quickly.  For example on my Suse Linux server here I was able to create a backup of a 750MB file in under 2 seconds using the CREATE FILE directive.  The system prevented any updates to the file during this process but still allowed other processes to have the file open and simply stalled any updates until the copy was complete. A few second delay during off-hours is likely not going to be noticed.

In theory you should be able to create a program that makes copies of all your files as your backup and optionally ZIP that that directory up or transfer to another machine.  This should help assure you clean backups.

[edjack]

Paul, have you considered using the CREATE FILE directive to make your backups?
https://manual.pvxplus.com/page/directives/create_file.htm

This directive (added in PxPlus 2016) was specifically designed to help with this problem.  It does a high-speed data backup of active data files while placing a temporary lock on the file that prevents any updates. 

Assuming your files are a reasonable size you should be able create duplicate files fairly quickly.  For example on my Suse Linux server here I was able to create a backup of a 750MB file in under 2 seconds using the CREATE FILE directive.  The system prevented any updates to the file during this process but still allowed other processes to have the file open and simply stalled any updates until the copy was complete. A few second delay during off-hours is likely not going to be noticed.

In theory you should be able to create a program that makes copies of all your files as your backup and optionally ZIP that that directory up or transfer to another machine.  This should help assure you clean backups.

Trying to get the syntax of the command working with a channel number.
Can someone post a sample of the "CREATE FILE" command using a channle number.
Thanks

[Paul Robinson]

The files detected as corrupt range in size from <1MB in size to in excess of 26GB. In the limited testing done with client backup media they have been consistently corrupt. I have not tried CREATE_FILE but will investigate. The client is on version PxPlus 12.5 but we can look into upgrading them to PxPlus 2016 if this offers a solution.

[Devon Austen]

Ed,

Here is an example of the CREATE FILE directive with a channel.

Code Select Expand


OPEN (1, iol=*) "clients"
CREATE FILE "clients.backup" FROM (1)


[Mike King]

Paul, a 26GB file should be able to be copied in about a minute or so during which time the users sessions will be suspended.
I would however suggest you rethink updating such large files in a live 7/24 hour environment. 

Here is my $0.02 (CDN) for what its worth...

My guess is that your 26GB file contains not only active information but also historical records such as past invoices or GL transactions.   If so, this means you are constantly backing up large amounts of data that doesn't change simply because it is intermixed on the files with active/changing data.

I generally suggest for systems where up time is critical that you try to keep active files small and move historical data to larger secondary files. 

The main reason I suggest this is that should the system fail due to hardware, power, or software failure, recovery times for large files can be excessive.  While you can somewhat mitigate this by adding commit logic, disc drive failures or similar hardware failures still can cause data corruption requiring full file recovery.

Ideally you should place historical 'fixed' data into files that are not updated during normal operations.  This way these files and their data won't need to backed up every day and recovery of the historical data is simply a matter of doing a restore.

This approach not only makes it easier/faster to recover your system in case of a file system failure but also allows you to speed up the time required to backup the data as you only need to backup data that is changing.
A few releases ago we added the ability to have PxPlus automatically split large files into individual smaller physical files in order to help clients accomplish this.

[Paul Robinson]

Hi Mike,
You are correct in your assumption the 26GB file contains both current and historical data. We are discussing the idea of archiving records with the client in an effort to make these files smaller. For us, access to these archived files is the challenge. It means coding changes to support another file set. I did read your line about having PxPlus auto-split large files. Not sure how that might work since it sounds like it's based on file size rather than date dependent. Our files are primarily accounting in nature so some control over the file split would likely be important.

[Mike King]

The file splitting is based on data.  Basically you can tell the system that records with some field (such as a date field) having a specific values (min/max range) go into a different physical file.  This allows you to setup individual files per year or even by month.

For example assuming you have a InvoiceDate field with the creation date of an invoice (which obviously does not change) you could have individual files for each year or even for each month and year.  If you start getting too many files, you could have some files split on the year with other split by month in the current year.

About the only issue you need to consider is how to determine when to consider the records permanent and no longer subject to change so the file can be locked and a permanent archive created.  For example will you want to allow an invoice created 2 years ago to be updated and thus force that years physical file(s) to be re-archived?

GL posting records are not normally an issue (auditors don't like it when you go back and tweak these ).