You can do this by:
creating a polygon based on data frame properties then
using Select Layer by Location (arcpy.SelectLayerByLocation_management)
(Much thanks to the poster at Geonet who gave me this idea:
Once you have panned and zoomed in your mxd to the area of interest, hit save (this updates the mxd and data frame objects to reflect your current view).
Here is the Python code that makes the magic happen:
#Get map document object (ensure map document is open and the right data frame is active)
mxd = arcpy.mapping.MapDocument(yourMXD)
#Get data frame object
df = arcpy.mapping.ListDataFrames(mxd, "*")
#Get df's spatial reference object
spatialRef = df.spatialReference
#Get data frame extent
frameExtent = df.extent
# Process: create new Visible Extent Polygon feature class to use to clip from your dataset.
arcpy.CreateFeatureclass_management(inWorkspace, "Visible_Extent_Polygon", "POLYGON", "", "DISABLED", "DISABLED", "", "", "0", "0", "0")
VisExtPoly = os.path.join(inWorkspace,"Visible_Extent_Polygon")
# Process: create a current visible extent polygon and copy it into the new feature class.
# This section of code takes the frameExtent, creates an array, and then converts the array to a polygon feature class.
XMAX = frameExtent.XMax
XMIN = frameExtent.XMin
YMAX = frameExtent.YMax
YMIN = frameExtent.YMin
pnt1 = arcpy.Point(XMIN, YMIN)
pnt2 = arcpy.Point(XMIN, YMAX)
pnt3 = arcpy.Point(XMAX, YMAX)
pnt4 = arcpy.Point(XMAX, YMIN)
array = arcpy.Array()
polygon = arcpy.Polygon(array) #creates polygon object
#Process: make feature layer so the visible extent polygon can be used as an input to Select Layer by Location.
# Process: Make Feature Layer for your large dataset of interest (in my case, a roads network) to be selected by visible extent
# Process: Select Layer By Location - FTEN roads
arcpy.SelectLayerByLocation_management("FTEN_roads_fl", "INTERSECT", VisExtPoly, "", "NEW_SELECTION", "NOT_INVERT")
# Process: Copy selected FTEN roads to FC, then clear the selection
arcpy.CopyFeatures_management("FTEN_roads_fl", FTEN_in_extent, "", "0", "0", "0") #creates the actual feature class in the .gdb
That works for one layer; you can use the Visible_extent_polygon to run a Select by Location ion any other layers you might have also.
The only bug I have in this code is that the spatialReference is not brought into the Visibel Extent Polygon layer, so it is drawn rather than projected in the map. But it still works as a selection layer.