Update: there's now a better solution to this, see here:
QGIS expression to access page-number of atlas for certain features
Just an idea for a partial workaround so that you don't have to do everything manually: insert on your atlas sheet an attribute table and list there the names of the towns (no. 1 in the screenshot below).In the feature properties of the table select the object filtering tab and activate the checkbox "Show only features visible on the map" (2). Add an additional attribute to the table in the print composer (3), this time a custon defined, based on the variable @atlas_featernumber - manually (3a) or in the expression editor (3b) - this should correspond to the page-no. If you navigate from page to page in the atlas, the variable will be printed as the no. of the current feature.
So if your first page in the print layout contains the map that generates the atlas, the variable is identical with the page-no. (otherwise add the amount of additional pages to the expression generating your manually created attribute in the edior, 3b). By this, you get for each atlas page a list of the towns showed on this page and in a second row the page-no (4). Copy all the lists and paste them together and you have your index.
To sort the list alphabetically, add a definition in the attributes dialogue window (underneath of 3a).
There might be a better solution based on the expression
intersects( $geometry , @atlas_geometry ) - thus creating a statement based on the question if the geometry of a town lies within (intersects) with the grid of your polyon-layer that generates the atlas-pages - however right now I don't know to implement the page-no. with this on a separate (empty) page to create a completely automatic index.