GRID_MESH/IRREGULAR_POLYGON_LGCP/IRREGPOLLGCP_OUTPUT DIRECTORY.

THIS DIRECTORY CONTAINS THE OUTPUTS FROM THE TWO SIMULATION STUDIES FOR THE IRREGULAR POLYGON LGCP IMPLEMENTATION OF THE GRID-MESH OPTIMISATION METHOD WHICH CONTAINS SEVERAL OUTPUT FOR EACH SIMULATION STUDY AS THEY WERE SPREAD ACROSS MULTIPLE NODES AND THE THEIR PROCESSORS, AND THEN THE OUTPUTS FOR EACH SIMULATION STUDY ARE COMBINED INTO A SINGLE OUTPUT WITH THE R SCRIPT IN GRID_MESH/BalenaOutputCombined_final.R.
ADDITIONALLY, THERE WERE SEVERAL PROCESSES WITH ERRORS, SUCH AS THE TIME ERROR OR SPACE ERROR AND SO, THE INITIAL COMPLETED RUNS FROM BALENA MAY HAVE HAD THESE ERRORS PRESENT AND REQUIRED A RE-RUN, THE OUTPUTS WITH SUFFIXES WERE THESE INITIALY OUTPUT FILES THAT WERE THEN RE-RUN ON BALENA WITH MORE PROCESSORS AVAILABLE. THESE FINAL OUTPUTS, WITH ERRORS RE-RUN WERE COMBINED FOR THE FINAL SINGLE OUTPUT FILE.

WE HAVE ADDITIONAL SUB-DIRECTORIES PERTAINING TO RUNS WHICH RESULTED IN ERRORS, WITH THE DESCRIPTIONS FOR THESE SUB-DIRECTORIES BELOW.

- GridMeshIrregPolLGCPTradSSi.rda: these are the outputs for the traditional simulation study which are combined in to GridMeshIrregPolLGCPTradSS.rda.

- GridMeshIrregPolLGCPSBCSSi.rda: these are the outputs for the SBC simulation study which are combined in to GridMeshIrregPolLGCPSBCSS.rda.

- GridMeshIrregPolLGCPSBCSSi_TIMEERRORFINAL.rda: these are the completed outputs before the time errors were re-run with a larger number of processors. These are then moved over to Balena to be loaded and have the necessary time errors re-run. These are produced within the GRID_MESH/IRREGULAR_POLYGON_LGCP/IRREGPOLLGCP_CODE/TimingErrorProcessandDataGeneration_final.R script.

- GridMeshIrregPolLGCPSBCSSi_FULL.rda: the outputs from the time error re-runs, although maybe the re-run took over 12 hours (the time limit) and while others were manually stopped these runs had been missed, and so for consistency we find such runs and re-set those particular outputs to be missing as though they were not retrieved initially. These data sets are the outputs containing these runs, before the output was removed and can also be found in the GRID_MESH/IRREGULAR_POLYGON_LGCP/IRREGPOLLGCP_OUTPUT/IRREGPOLLGCP_OUTPUT_ERROR directory as well.

- GridMeshIrregPolLGCPSBCSSi_SPACEERRORFINAL.rda: these are the completed outputs (inc. any re-runs for time errors having been completed) before the errors due to no space for memory allocation were re-run. These are then copied over to Balena for loading and re-running.

- TimingErrorDataFrames.rda: this is the output from GRID_MESH/IRREGULAR_POLYGON_LGCP/IRREGPOLLGCP_CODE/TimingErrorProcessandDataGeneration_final.R and contains a list of data frames for each process that had at least one timing error where the data frame contains information on the simulation-grid-mesh indices as well as the index for the error that will be found in the error message which is important for matching the correct simulated data (stored in the sub-directories TimeErrorData/TIMEERRORj) to load for the error re-run. Finally, there is a column in the data frame which is a numeric code, initialised to zero, to tell whether a re-run has been completed (1), re-run produced and error (2), or not re-run (0). 

- ProcessiErrorDf.rda: these are individual data frames from the list TimingErrorDataFrames.rda for each process with at least one error. This output is saved for each individual process during the re-runs for easy re-loading if a run is needed to re-start due to any time limits. These are the same as the data frames in TimingErrorDataFrames.rda but with the final column indicating re-runs completed/errors/not started filled in as the time error re-runs occur. These data frame outputs for each process, after the TIME ERROR re-runs, should have a 1 in the final column if the run was complete, if there is a 2 then there was an error when running, usually this also occurred after 12 hours (or the error was the > 12hr run) and so was not re-run. Additionally, in our case there are two with the value 0, which originally indicates that this was not re-run but in our case, the re-run took place but took over 12 hours although when we manually stopped the run, there was no 2 placed in the final column as it should have been.

THE SUB-DRECTORY IRRREGPOLLGCP_OUTPUT_ERROR CONTAINS THE R CODE WHICH WILL INSERT THE TIME/SPACE ERRORS IN ORDER TO ALLOW THE REMAINING SIMULATIONS AND COMBINATIONS TO FINISH BEFORE RE-RUNNING THE TIME/SPACE ERROR WITH MORE PROCESSORS. THIS SUB-DIRECTORY CONTAINS ITS OWN README FILE.

INITIALLY, DUE TO TEMPORARY DIRECTORY ISSUES, THERE WERE JOBS THAT FAILED WITH A MESSAGE ABOUT THERE NOT BEING ENOUGH MEMORY LEFT, OFTEN THIS PRODUCED ERRORS IN BOTH OF THE PROCESSES RUNNING ON THAT NODE, WHILE RARELY IT PRODUCED AN ERROR IN JUST ONE. IF THERE WAS AN ERROR IN BOTH PROCESSES, THEN IN ORDER TO DEDUCE WHICH HAD CAUSED THIS ODD ERROR, THE ERRORS (LABELLED SPACEERROR(2)) WERE REMOVED (RE-SET) WITH IrregSBCErrReSetProcs_final.R AND RE-RUN. IF THE SPACE ERROR RE-OCCURRED (WHICH WAS NOT ALWAYS THE CASE) THEN COMPARING THE SIMULATION-GRID-MESH COMBINATIONS AT THE TIME OF THE ERROR INDICATED WHICH RUN WAS `TO BLAME', AS IT WOULD HAVE BEEN THE SAME LOCATION AS THE PREVIOUS ERROR. THEREFORE, THE ERRORS IN THE OTHER RUNS WERE REMOVED WITH IrregSBCErrRemovalProcs_final.R WHILE THE ERRORS IN THE OTHERS WERE RETAINED. AFTER THE ISSUE WITH THE TEMPORARY DIRECTORY WAS SOLVED, THESE ERRORS WERE RE-RUN (FROM *_SPACEERRORFINAL.rda) AND THE ERRORS REMOVED WHEN COMPLETED. IF THERE ARE NO TEMPORARY DIRECTORY ISSUES THEN THESE SPACE ERRORS SHOULD NOT OCCUR.

IN ADDITION TO THESE SPACE ERRORS, WE HAD THE TIME ERRORS PRESENT, AND THESE OCCURRED FOR MULTIPLE PROCESSES AND, FOR SOME OF THESE PROCESSES, THEY OCCURRED MULTIPLE TIMES ACROSS THE SIMULATIONS. WHEN A RUN TOOK TOO LONG (>6hrs), WE PAUSED THE RUNS AND INSERTED A `TIME ERROR j' MESSAGE IN THE OUTPUT (WHERE j INDEXES THE TIME ERRORS FOR THE PROCESSES THAT OCCURRED AROUND THE SAME TIME) SO THAT WE COULD CONTINUE RUNNING THE OTHER SIMULATIONS AND RETURN TO THESE TIME ERRORS LATER.

NOTE THAT FOR THE DATASETS IN THIS SUB-DIRECTORY, CAPITALS WITH RESPECT TO THE ERROR SUFFIX INDICATE THAT THIS IS THE OUTPUT AT THE TIME IT WAS REMOVED FROM BALENA DUE TO AN ERROR AND WILL BE THE INPUT TO ONE OF THE FUNCTIONS BELOW, WHILE SUFFIXES NOT OF THIS FORM (I.E. *_PostErrResetj.rda) ARE USUALLY THE OUTPUTS FROM THESE FUNCTIONS THAT ARE RE-LABELLED TO STORE AS BACK-UPS.

FOR EACH OF THE ERROR INSERTION/REMOVAL R SCRIPTS THE OUTPUT IS ORIGINALLY GridMeshIrregPolLGCPSBCSSi.rda AND ARE IMMEDIATELY MOVED BACK TO BALENA TO CONTINUE THE SIMULATION STUDY, HOWEVER (AS NOTED BELOW) BACK-UPS OF THESE ARE ALSO CREATED, AS RUNNING ANY OF THESE R SCRIPTS AGAIN FOR ANOTHER ERROR WILL OVER-WRITE THESE OUTPUTS.

SPACE ERROR FUNCTIONS:
- IrregSBCErrReSetProcs_final.R: this code is to re-set the errors to NA in order to re-run and see which run caused the initial error. In some cases, both runs passed the original error position for simulation-grid-mesh and so were left to continue running. We would expect that this should not be required if the simulation study has no issue with the temporary directory.
	-- Input:
		--- GridMeshIrregPolLGCPSBCSSi_SPACEERRORj.rda: these outputs (where j is empty or 2) were the results of the memory error and the errors were re-set with IrregSBCErrReSetProcs_final.R which outputs GridMeshIrregPolLGCPSBCSSi.rda which are returned to Balena to continue running. The j=2 distinguishes the space errors that were caught later during the simulation study than the other processes, after the original space errors had been reset and were continuing to run.
	-- Output: (in addition to GridMeshIrregPolLGCPSBCSSi.rda)
		--- GridMeshIrregPolLGCPSBCSSi_PostErrResetj.rda: these are the outputs created from IrregSBCErrReSetProcs_final.R where j is either nothing to 2, and also saved under this name in the case of the error re-occurring and so preventing the new GridMeshIrregPolLGCPSBCSSi.rda being copied over and overiding the original output of the reset. Even if there are no new errors and the re-created output in this sub-directory is not overwritten, it is interesting to have a back up of the output at this stage.

- IrregSBCErrRemovalProcs_final.R: this code is to remove the errors (replace with NAs) for the runs that surpassed the previous simulation-grid-mesh settings when the original space errors occurred and therefore were not the `cause' of the errors. By removing these errors they can continue without errors that only occurred as the job failed due to the other run. We would expect that this should not be required if the simulation study has no issue with the temporary directory.
	-- Input:
		--- GridMeshIrregPolLGCPSBCSSi_REMERROR.rda: these outputs were renamed from GridMeshIrregPolLGCPSBCSSi.rda outputs from Balena in order for IrregSBCErrRemovalProcs_final.R to remove the errors that were caused by the memory issue but not related to these processes in particular so they can be re-set and continue to run as normal. These files are taken in and GridMeshIrregPolLGCPSBCSSi.rda are output.
	-- Output: (in addition to GridMeshIrregPolLGCPSBCSSi.rda)
		--- GridMeshIrregPolLGCPSBCSSi_PostRemErr.rda: these are the outputs created from IrregSBCErrRemovalProcs_final.R, and also saved under this name (as back-ups) in the case of the another error re-occurring and so preventing the new GridMeshIrregPolLGCPSBCSSi.rda being copied over and overiding the original output of the error removal.

TIME ERROR FUNCTIONS:
- IrregSBCTimeErrInsertProcs_final.R: this code allows us to insert TIME ERRORS in to the SBC outputs and re-save the outputs so that we can continue to run the simulation study so that the particular simulation-grid-mesh combinations that have time errors can be returned to in order to re-run with a higher number of cores available.
	-- Input:
		--- GridMeshIrregPolLGCPSBCSSi_TIMEERRORj.rda: these were the outputs where the runs took over 6 hours (or an unrealistically long time) without any result and therefore has `TIME ERROR j' placed in the required simulation-grid-mesh error slot, with the first Time Error just denoted `TIME ERROR'. For j=2, some errors were called *_TIMEERROR2SKIP.rda due to the grid-mesh combination intended to run after the TIME ERROR 2 addition having a much larger mesh resolution which could also have resulted in another time error and therefore, these errors were pre-emptively included so as not to risk waiting another 6 hours where the computations could likely not be completed which would require another manual stop. The incrementing index j for the time errors were mostly due to tracking multiple time errors through the same processes and so needing to distinguish between the errors, especially for re-running. Additionally, as the time errors occurred at different times throughout the simulation study, in order to keep track it was easier to group the processes that had time errors around the same time together with the index j, and the next time processes had time errors we group them together and increment j. These outputs are taken in by IrregSBCTimeErrInsertProcs_final.R where GridMeshIrregPolLGCPSBCSSi.rda is output. (There are additional outputs for processes 5 and 14 where TIME ERROR 5 was incorrectly labelled as TIME ERROR 4, and this was also corrected in IrregSBCTimeErrInsertProcs_final.R).
	-- Output: (in addition to GridMeshIrregPolLGCPSBCSSi.rda)
		--- GridMeshIrregPolLGCPSBCSSi_PostTEj.rda: if the same process resulted in another time error, then the previous output from IrregSBCTimeErrInsertProcs_final.R are re-named with this additional suffix, in order to retain the output from the addition of TIME ERROR j as a back-up in case it may be needed in the future.

TIME-ERROR POST-RUN TIMING CHECKS
- TimeErrorsTimeChecks_final.R: if the results from the Time Error re-runs finished in over 12 hours, which may not have initially been caught in order to manually stop the runs, then the outputs were removed, to ensure consistency across all processes, where the 12 hour time limit was set for the additional re-runs. This also takes in the relevant (manually relabelled) *_FULL.rda outputs for these processes and removes the relevant outputs that took over 12 hours to complete.
	-- Input:
		--- GridMeshIrregPolLGCPSBCSSk.rda: these are the all of the outputs for the SBC simulation study after the time error re-runs were completed, however some runs took over 12 hours and were not initially caught (as for some of the runs, whch had been stopped before they could run for an excessively long time), we therefore read these files in to identify the relevant files and manually relabel these outputs with the suffix "_FULL", mentioned below.
		--- GridMeshIrregPolLGCPSBCSSi_FULL.rda: the outputs from the time error re-runs where the time error re-run took over 12 hours (the time limit for the re-runs) and while others were manually stopped these runs had been missed. Therefore, for consistency we find such runs and re-set those particular outputs to be missing as though they were not produced to being with. These data sets are the outputs containing these results, before the relevant output was removed and can also be found in the GRID_MESH/IRREGULAR_POLYGON_LGCP/IRREGPOLLGCP_OUTPUT directory as well but note that the code to reset the data is implemented in this sub-directory.
	-- Output:
		--- longrunstimeerror.rda: this contains a data frame with the processes that, when re-run with 16 processors on a node, took over 12 hours to re-run, and so these GridMeshIrregPolLGCPSBCSSi.rda are manually re-labelled as GridMeshIrregPolLGCPSBCSSi_FULL.rda and then the remainder of this R script is run and the relevant time-consuming outputs are removed.
		--- GridMeshIrregPolLGCPSBCSSi.rda: the final output for process i after the necessary error re-runs and this will be moved to the main directory, IRREGPOLLGCP_OUTPUT to be combined with the outputs from the other processes.

- ComparingNewandOldProcs_final.R: this takes the outputs from the final pre-error re-run outputs and compares them to the outputs post-space-and-time-error re-runs to ensure there's is nothing drastically wrong with the results.

- *.txt: these are the various outputs from the reset/removal/insertion of errors to check that the R scripts for any of these did not return odd outputs and so they can be replaced on Balena for further runs.

THE SUB-DIRECTORY TimeErrorData CONTAINS INDIVIDUAL SUB-DIRECTORIES LABELLED TIMEERRORj WHERE j=1,...8 AND CONTAIN THE DATA FRAMES THAT RESULTED IN THE TIME ERRORS, THESE WERE STORED FOR THE RE-RUNS OF THE ERRORS TO PREVENT THE NEED TO RE-SIMULATE THESE DATA SETS. THIS SUB-DIRECTORY DOES NOT CONTAIN ITS OWN README FILE.

FINALLY, THE SBCCompletedPre-TimeErrorReruns SUB-DIRECTORY CONTAINS THE OUTPUTS FROM THE SBC SIMULATION STUDY BEFORE ANY RE-RUNS WERE COMPLETED, FOR EITHER THE TIME OR SPACE ERRORS. IN ADDITION TO THE FINAL OUTPUTS WITH THE ERRORS STILL PRESENT, WE HAVE THE COMBINED OUTPUT FILE GridMeshIrregPolLGCPSBCSS.rda FROM THESE TIME ERROR-FILLED INDIVIDUAL OUTPUTS. THIS SUB-DIRECTORY DOES NOT CONTAIN ITS OWN README FILE.
