If your goal is to create a polygon representing the outline of a set of features, you're probably looking to create the concave-hull of this features set.
While the creation of a convex-hull is a pretty straightforward operation, because there is only one possible convex-hull for a given set, this is not true in the case of the concave-hull.
Because concave-hull polygons are allowed to have angles greater than 180º, their creation can have more than one possible solution.
The QGIS Concave Hull plugin uses a nearest-neighborhood approach to allow the user define the "concaveness" of the output polygon.
You can see more details about the plug-in, as well as different possible output polygons for the same feature set in the plugin GitHub page.
It was written based on the algorithm described by A. Moreira and M. Y. Santos (2007): Concave Hull: A k-nearest neighbors approach for the computation of the region occupied by a set of points
Other open-source GIS implementations of convex-hull computations are available on PostGIS (ST_ConcaveHull) and GRASS GIS (v.concave.hull).