I know that NED DEM layers are created using mainly Lidar point clouds and that the SRTM derives its data from shuttle-based radar. I have both of these loaded into QGIS and can see that over my study area of the Tampa metro area the two layers differ quite substantially in some areas. I have read that the SRTM will sometimes read the tops of ground features like forests as the ground elevation. This would hurt my research because I am only interested in actual ground elevation. My issue with the NED is that it sometime appears to have a measurement that is below what I would expect the elevation to be, especially around the coast. Is either one of these elevation layers a better source for finding actual surface elevation?
This is by no means an answer to your specific problem but an explanation of elevation data and vertical accuracy. Also, this all wouldn't fit in a comment.
Only a select number (depending of area of lidar coverage) of elevation control points are used when checking the vertical accuracy of lidar data. I wouldn't expect all elevation markers to match elevation data, but 1-3 feet does seem more than would be expected. However, consider USGS Base Specification absolute vertical accuracy standards associated with varisous quality levels. This document is our bible when calibrating and classifying lidar point clouds, as well as generating derivatives such as DEMs for the USGS.
QL3 raster data has a minimum cell size of 2m. If you are using a 3m NED DEM, we can consider this a QL3 raster. The minimum vertical accuracy standards for a QL3 lidar dataset is an RMSEz of less than or equal to 20cm (7.9in), and an NVA at 95th percentile confidence level of less than or equal to 39.2cm (15.4in). This meaning that you can be 95% confident that any lidar point is 39.2cm above or below it's real world location.
Also, consider the raster as a data structure. The raster aggregates points to a regular grid, and represents that aggregate area (3 square meters in the NED case) as a single value, usually where the centroid falls on the TIN that was used to create the DEM. So if you are comparing a raster value to a single point, especially one on a slope, consider these things that introduce error into the data (remember no data is without error). I am sure there are vertical accuracy standards for the SRTM data as well. The pros and cons of elevation data should be considered in any sort of analysis.