11

I have Apache running Geoserver on my virtual instance, and I am wondering whether the Geoserver data directory can point to an Amazon S3 bucket. Is this possible? How does this affect performance?

If S3 is a poor setup for Geoserver's data directory, are there any suggestions for adding large amounts of data to my Geoserver data directory? The data directory may reach up to 100's of GB's, possibly more. Should I just get a server with a large hard drive, or connect to a more scalable solution, like an Amazon S3 bucket?

  • What is the data? I guess imagery, but have you tiled it? Maybe some database for the raster win :-) – BradHards Feb 13 '15 at 11:04
  • 1
    If you data is raster and will be read and processed for interactive WMS S3 might be dead slow. Hints from past experiences, you might want to look at glusterfs + EBS. – simogeo Feb 13 '15 at 11:12
  • 1
    I have an interest in using S3 as a shared data bucket for a horizontally elastically scalable fleet of GeoServers. I am currently investigating the use of S3Fuse to mount a file system for this. It would be great to hear of any other successes/failures in this area. – 8bitjunkie Feb 19 '15 at 9:26
  • Any experiences in this subject ? – geogeek May 25 '15 at 20:16
9

Pointing "directly" to an S3 bucket (e.g. using s3fs) is "probably" a bad idea. e.g. https://stackoverflow.com/questions/10801158/how-stable-is-s3fs-to-mount-an-amazon-s3-bucket-as-a-local-directory

EBS is almost certainly the right place to store the data for access by GeoServer.

100 GB is very manageably by EBS standards. (Maximum volume size is 16 Terabytes!)

I might suggest to store the original data in S3 and then replicate onto your instances as needed, using AWS CLI tools to sync the contents of an S3 folder to your local instance with a single command. http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html). S3 is arguably redundant, but I find that it adds flexibility and convenience.

Side note: If you are planning to use large multi-core instances anyway AND you know the data volume is not likely to grow unexpectedly AND you are able to auto-build your instances, you could actually consider using instance storage ("ephemeral storage") because your original data will be safe in S3, you can save on EBS fees, and instance storage can be slightly faster than EBS in some cases.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.