0

I am trying to retrieve data from 324 eopatches I sampled beforehand. Patches are basically numpy arrays as explained in this documentation EOPatch. This is an example of the content of a patch after I resampled it :

EOPatch(
  data: {
    FEATURES: FeatureIO(/data/FEATURES.npy)
    FEATURES_SAMPLED: FeatureIO(/data/FEATURES_SAMPLED.npy)
  }
  mask: {}
  scalar: {}
  label: {}
  vector: {}
  data_timeless: {}
  mask_timeless: {
    LULC: FeatureIO(/mask_timeless/LULC.npy)
    LULC_ERODED: FeatureIO(/mask_timeless/LULC_ERODED.npy)
    LULC_ERODED_SAMPLED: FeatureIO(/mask_timeless/LULC_ERODED_SAMPLED.npy)
  }
  scalar_timeless: {}
  label_timeless: {}
  vector_timeless: {}
  meta_info: {}
  bbox: BBox(((500077.09501641133, 5095402.981379905), (501746.98615037295, 5097085.523204274)), crs=CRS('32633'))
  timestamp: [datetime.datetime(2017, 1, 1, 0, 0), ..., datetime.datetime(2017, 12, 19, 0, 0)], length=23
)

To retrieve certain data from the patch as my training and testing data. I first loaded the 324 patches like this:

# load sampled eopatches
eopatches = []
path_out_sampled = './eopatches_sampled_Slovenia'

for idx in range(len(patchIDs)):
    eopatches.append(EOPatch.load(f'{path_out_sampled}/eopatch_{idx}', lazy_loading=True))

eopatches = np.array(eopatches)

And then I used :

labels_train = np.array([eopatch.mask_timeless['LULC_ERODED_SAMPLED'] for eopatch in eopatches[patchIDs]])
features_train = np.array([eopatch.data['FEATURES_SAMPLED'] for eopatch in eopatches[patchIDs]])

However, it was taking forever executing and it was taking up so much RAM.

patchIDs is a list of numbers ranging from 0 to 323.

I tried to use this instead:

def data_retrieval(eopatch):
  feature_data=[]
  label_data=[]
  for eopatch in eopatches:
    f=eopatch.data['FEATURES_SAMPLED']
    feature_data.append(f)
    l=eopatch.mask_timeless['LULC_ERODED_SAMPLED']
    label_data.append(l)
  feature_data=np.array(feature_data)
  label_data=np.array(label_data)
  return feature_data, label_data

I just ended up with the same result.

enter image description here

I have also reinitialized the environment multiple times but nothing changed and I am using GPU. Using h5py didn't help as well. I was able to load only 20 patches out of the 324.

  • Do you know how much memory the process is taking when it is executed? If it's exhausting your physical RAM and forcing the OS to swap to and from disk, that could slow the whole thing down. On the other hand, if you've got enough RAM, then perhaps it's time to break out cProfile and determine which calls are taking the most time: https://docs.python.org/3/library/profile.html – Modern geoSystems May 21 at 19:16
  • The whole code runs smoothly until I execute one of those two lines : labels_train = np.array([eopatch.mask_timeless['LULC_ERODED_SAMPLED'] for eopatch in eopatches[patchIDs]]) features_train = np.array([eopatch.data['FEATURES_SAMPLED'] for eopatch in eopatches[patchIDs]]) where the RAM instantly gets taking up – Rim Sleimi May 21 at 19:39
  • Ah! Have you considered using something like h5py? It'll definitely use a disk cache, but likely more efficiently than relying on virtual memory. – Modern geoSystems May 21 at 19:50
  • What I can't understand is that I did upload the whole 324 patches before that and it didn't take up much of RAM but now that I'm extracting a subset of each patch the RAM instantly hits its limits – Rim Sleimi May 21 at 21:13
  • Oh, that's new information. Not being familiar with your data, I kinda assumed it was bulky overall. – Modern geoSystems May 21 at 22:37
0

I suspect that you really have a large dataset underneath your patch objects, and the lazy loading is masking that until you start to retrieve specific subsets. I suggest you use a tool more suited to large datasets, such as h5py. Hopefully that or a similar tool will still meet your requirements.

| improve this answer | |
  • I did use h5py but it failed to load as well – Rim Sleimi May 22 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.