"Sheer" means:

"The curved fore and aft line formed by the uppermost plank of a boat"

So the sheer plank is the plank of wood that runs along the outside of the top of the hull on either side. You might call it the rim of the hull. So launching a whale-boat from the sheer-plank means launching it from the top of the side of the ship.

Here's a picture of the sheer plank being mounted on a traditional wooden ship:
http://bigboatbuild.com/big-boat-build-gallery.php?galleryid...
And here's a video of it being done:
http://bristolboatcompany.com/post/11258342412/h20-sheer-pla...

--------------------------------------------------
Note added at 23 mins (2012-05-29 12:41:13 GMT)
--------------------------------------------------

I forgot to add the reference for the definition of "sheer"; it is:
http://www.seatalk.info/cgi-bin/nautical-marine-sailing-dict...

Here's a definition of "plank sheer", which I think means the same:

"Plank Sheer.
Plank resting on top timbers of ribs or frames"
http://www.scribd.com/doc/44215372/Nautical-Dictionary#outer...

"The Oseberg ship was both rowed and sailed. The sheer plank on either side has 15 ports for oars"
http://nautarch.tamu.edu/class/316/oseberg/
On a Viking ship (illustrated). The ports for oars are obviously at the top of the sides of the ship; they have to be in order to be able to row.

"Paint-strake, or Sheer-plank, the uppermost strake of plank on the vessel, terminating the sheer or vertical curve of the top sides"
A Treatise on Marine Architecture (1830)
http://books.google.es/books?id=DQBQAAAAYAAJ&pg=PA116&lpg=PA...

(The trouble with nautical dictionaries is that the definition is often as hard to understand as the word you're looking up.)

--------------------------------------------------
Note added at 45 mins (2012-05-29 13:03:28 GMT)
--------------------------------------------------

The launch of the whale-boat, done surreptiously at night to enable the narrator and his companions to escape from the ship, is described in chapter 8 of Mardi. The boat "hung from the ship's lee side, the side depressed in the water", so it must have been suspended by ropes from the sheer-plank. What made it so hazardous was that the ship was moving at the time: "the sole daring of the deed that night achieved, consisted in our lowering away while the ship yet clove the brine". Normally you'd bring the ship to a halt first.