When rendering a TMS layer, OpenLayers cuts the area you are currently viewing into rectangles of the size tilewidth x tileheight, so you should set the tilewidth and tileheight to the actual size of your tiles when you create the TMS layer.
Now, for every such rectangle in your viewport, it calls a function (usually named getTileUrl(bounds)), which receives a bounds object (specifying the rectangle bounds) as an argument and should return the corresponding tile's url (you are going to write that function and pass it as an argument when creating the TMS layer).
So, suppose your TMS layer is covering the map in such a way that the first tile is positioned exactly in the lower left corner of the map (its lower left corner is identical to the lower left corner of the map). Unless specified otherwise, OpenLayers supposes this is true (i.e. the tileOrigin point is identical to the lower left point of mapExtent). Furthermore, let's assume that the tiles are named for example R0C0.png, R0C1.png, R0C2.png for the first row of tiles, then R1C0.png, R1C2.png, R2C2.png for the second row etc.
Now, for a given bounds object that specifies the rectangle OpenLayers is asking about, how do you compute the name of the tile which should be there? You just need to find out what row is it in (let it be y), what column is it in (let it be x) and then you just return the name RyCx.png.
So, how to find out the x? The first tile should begin at the leftmost (westernmost) border of the map. The x coordinate (or longitude, according to your projection) of the leftmost border of your map is stored in this.mapExtent.left. This tile is this.tileSize.w wide in pixels, and if the resolution is res, it is this.tileSize.w * res geographic units wide on the map. By geographic units I mean degrees for example, or any other unit your projection uses. The resolution is the ratio geographic units on the map / pixels on the screen.
So, if the first tile starts at this.mapExtent.left and it is this.tileSize.w * res wide, then the next tile starts at this.mapExtent.left + this.tileSize.w * res, the next one at this.mapExtent.left + 2 * this.tileSize.w * res, another one at this.mapExtent.left + 3 * this.tileSize.w * res etc.
So, apparently, if you get the bounds of an area which corresponds to a tile and you want to know the x, i.e. in which column of the TMS grid the tile is, you use the formula
x = (bounds.left - this.maxExtent.left) / (res * this.tileSize.w)
Similarily, you use
y = (bounds.top - this.maxExtent.top) / (res * this.tileSize.h)
(or you can negate the difference in the numerator - it depends on whether the first row is the top row or the bottom row).
Eh... It's 2am here, so this could be far from comprehensible. Well... I just hope it will.