Without viewing the data I suspect that you have data in two different vertical datums.
The LiDAR data will almost certainly include a LiDAR report that will detail the vertical datum of the LiDAR derived DTM z units. This report will also include an accuracy assessment and give you some idea of the quality of these data. LiDAR accuracy will vary based on land cover, slope, reflectance, and of course the quality of the GNSS units and LiDAR equipment. Of the five LiDAR data I have available to me these surfaces vary from 5 - 15 cm from each other depending on location. In the USA these LiDAR data will likely reference the NAVD88 vertical datum.
Your photogrammetry data was probably derived from drone imagery and structure from motion software (SfM). These SfM derived surfaces will reference the elevation values from the drone GNSS unit unless you specifically tell the SfM to use a different vertical datum - and provide ground control points that reference that datum. Unless told otherwise the drone data would be in WGS84.
Sort out your vertical datums and report back. See this link for a vertical datum tutorial. Use a metadata reader, something like ArcCatalog, gdalinfo, or maybe EXIFtools to return some metadata from your surfaces and include those in your post. Use this post to see what the vertical datum differences might be for your area of interest and compare those results to your datasets.
Keep in mind that a consumer grade drone has a consumer grade GNSS unit. These GNSS units do not record elevations well. Converting your SfM surface values to the vertical datum values of the LiDAR will still result is elevation differences due to the poor elevation values of the drone.