You should use Route solver, the Closest facility wouldn't give you what you need.
You are supposed to have one or two point feature class(es) with the geometries and a field for route name. If you store both source points and destination points in one feature class, you need to load the source points first (so they will get sequence of 1). After that, load destinations (each point will get a sequence of 2). This can be done by using first the Select By Attributes (so, only selected rows will be loaded).
If you store them as separate feature classes, just load them one after another (loading the source points first).
When loading points, specify the field for
RouteName. If you have 100 source points and 100 destination points (that is, 100 pairs), 100 routes will be created for you in the Network Analyst window. Now you can solve the routes and you will get back information on the travel time and distance for each route.
More info about Route solver.