I had this same issue. After taking a closer look at my raster data (using gdal_info from the OSGeo4w shell), I discovered that there were very low negative minimum and very high maximum values, which I was not seeing in the metadata or histogram sections of the raster's properties in the QGIS table of contents. So, using gdal_calc, I eliminated the extremely low and high values through a couple expressions. My DEM had a natural maximum of 890 feet. And, I wanted to preserve nodata values as zero. So, the two expressions (using the OSGeo4w shell) were...
- gdal_calc --outfile="out_file_name_1.tif" -A "input_dem.tif" --calc="A*(A<890)"
- gdal_calc --outfile="out_file_name_2.tif" -A "out_file_name_1.tif" --calc="A*(A>0)"
Then, using "out_file_name_2.tif" as input into gdal_contour, it worked as expected.
When gdal_contour fails because of memory allocation, chances are either 1) your file is too large (which was not the case for me), 2) your contour interval is too small, or 3) the input raster has a large erroneous spike or trough causing excessive contours to be generated.
Hope this helps!