Of course you have considered a few approaches already?
One way is to think of the shape as an island. Then you calculate how many miles of shoreline it has, relative to its area. The more shoreline, the greater the roughness.
On this basis your shape at left scores low. The one at right scores high.
However this does not work for a narrow rectangle, which scores high yet its outline is not what you would call rough.
It seems that roughness is not an easy factor to quantify. When you think about it, the scale will be an arbitrary one. A perfect circle should score zero (or 1 depending on where you want to start). A circular saw disk might score 100.
However consider a fine tooth saw. It starts to resemble a smooth circle again. Therefore we should set a threshold as to how fine the scale should be. To put it another way, the scale ought to be proportional to the overall size of the shape.