If you discharge the battery pack down to 3.2 V, then that means each cell is 0.8 V each. This is about the point where we call the cell discharged.
Furthermore there will be a voltage drop through all devices (diodes, transistors, etc.) which you install to switch the power source, boost the power source, etc. These will cause a burden on battery power.
Furthermore the cells may not discharge evenly. One cell could go down to 0V, with the other cells at 1V. Then the one cell will start to be forced into reverse voltage. This is likely to result in oozing electrolyte, and shortening of the battery's useful life.
As an alternate, consider a single buck-boost converter. This is a switched-coil converter which can step-down or step-up, by altering the duty cycle. It would step down your 6V to 3.3V efficiently, and it would step up your 1.6V to 3.3V efficiently.