You can use GDAL's ogr2ogr utility (http://www.gdal.org/ogr2ogr.html) to clip a polygon by an extent. If you have GDAL for Python you can use something similar to this script to perform the task:
from osgeo import gdal
import subprocess
# define file paths
inRasterPath = 'C:/path/to/file.tif'
inVectorPath = 'C:/path/to/file.shp'
outVectorPath = 'C:/path/to/file.shp'
# get the extent of the raster
src = gdal.Open(inRasterPath)
ulx, xres, xskew, uly, yskew, yres = src.GetGeoTransform()
sizeX = src.RasterXSize * xres
sizeY = src.RasterYSize * yres
lrx = ulx + sizeX
lry = uly + sizeY
src = None
# format the extent coords
extent = '{0} {1} {2} {3}'.format(ulx, lry, lrx, uly);
# make clip command with ogr2ogr - default to shapefile format
cmd = 'ogr2ogr ' + outVectorPath + ' ' + inVectorPath + ' -clipsrc ' + extent
# call the command
subprocess.call(cmd, shell=True)
If you don't have GDAL for Python, you can use the OSGeo4W shell that comes with QGIS and run the gdalinfo
command on your raster to get the extent and then the ogr2ogr
command to clip the vector to the extent noted from the gdalinfo result with the -clipsrc
flag.
Make sure your polygon and raster file are in the same projection.