The Coast Guard spends much more time thinking about how to keep ships from sinking than it does about how to sink them. But because the Coast Guard is tasked with maritime security and because of the potential for terrorists using a ship as a means of attack, the question has become relevant. I don’t believe the 57 mm gun is adequate to stop a medium to large ship.
Stopping–keeping it from reaching the target–rather than sinking a ship is probably more the relevant criteria. Generally ships don't sink rapidly, particularly if you are trying to do it with a gun, so it is necessary to do enough damage to stop it in a timely fashion. Many of the ships that I will talk about continued to fight on for over an hour after the first hits were registered. Think of sinking a close surrogate for stopping a ship before it reaches its objective.
There are of course many examples of ships either surviving grievous attacks or alternately ships sink after a single hit. What it takes to sink a ship is highly variable and dependent upon ship design and preparation. But the most important variable seems to be size.
World War II experience
Over a long period, I've made an informal study of this subject. The amount of damage these little ships took and in some cases survived is truly amazing.
The US Navy Report of War Damage series briefly outlines all incidents of damage to US Navy Battleships, Carriers, Cruisers, Destroyers, and Destroyer Escorts as they were known at the time the document was published. Causes for sinkings were listed as follows:
• 38 by torpedoes alone (41.3 percent)
• 16 by suicide planes (17.4 percent )
• 12 Bombs alone (13 percent)
• 11 by gunfire alone (12 percent)
• 6 by torpedoes and gunfire (6.5 percent)
• 5 by mines (5.4 percent)
• 4 by torpedoes and bombs (4.3 percent)
• 1 by bombs finished off by gunfire (1.1 percent)
Torpedoes were involved in 48 of the 92 sinkings, or 52 percent.
As the ships get larger, it becomes harder to sink them by gunfire alone. If we consider only the larger major warships (Battleships, Carriers, Cruisers) lost, torpedoes were involved in sinking 74 percent, including 100 percent of the battleships and fleet carriers. Torpedoes were also involved in the sinking of all six battleships at Pearl Harbor.
• 10 by torpedo alone (43.5 percent)
• 4 by torpedo and gunfire (17.4 percent)
• 3 by bombs and torpedoes (13 percent)
• 3 by suicide plane (13 percent)
• 2 by gunfire alone (8.7 percent)
•1 by bombs alone (4.3 percent)
The two large ships sunk by gunfire alone both engaged heavy cruisers and/or battleships. USS Astoria (CA34) was sunk at the Battle of Savo Is., 1942. She was hit at least 65 times by 8-inch and 5-inch projectiles and took five 5-inch hits below water line. In spite of uncontrolled fires, she did not sink until a magazine exploded nine hours after the action. The opposing force was 5 CA, 2 CL, 1 DD.
USS Gambier Bay (CVE73), was sunk at the battle of Samar in 1944. She was hit over 26 times. Two projectiles penetrated shell plating below waterline and detonated in forward engineroom and after engineroom, respectively. Hits included battleship caliber rounds, possibly including hits by Yamato’s 18.1-inch guns. The opposing force was 4 BB, 6 CA, 2 CL, 11 DD. Notice in both cases, shells penetrated below the waterline.
As a point of reference, 57 mm projectiles weigh about six pounds, a 76 mm 12-14 pounds, a destroyer’s 5-inch 55 pounds, a light cruiser’s 6-inch (152 mm) 110 pounds, a heavy cruiser's 8-inch (205 mm) 260 pounds, an smaller battleship’s 14-inch, the Yamato’s 18.1-inch 3219 pounds).
Only one ship appears to have definitely been destroyed by gunfire from weapons 5 inches or less, the Longshaw (DD559) hit six times while aground off Okinawa, resulting in a magazine explosion that blew off the bow. No ship larger than 3,000 tons full load was sunk by gunfire from weapons 5 inches or smaller.
Consider that all these ships, including the aircraft carriers and battleships, were smaller than merchant ships that are now common. The largest was the carrier Lexington (CV2), 43,055 tons (fl), 888 ft (oa), 105’5″ beam, 33’4″ draft (270.66 x 32.12 x 10.15 m) (She was destroyed as a result of the accumulation of gasoline vapors after two torpedo and two bomb hits. The direct result of the hits were relatively minor, it was the gasoline vapor explosion that destroyed the ship). Lexington was a big ship, but no longer remarkable.
Warships might be thought more immune to damage. They usually have the advantage of better compartmentalization, a larger crew for firefighting, and sometimes armor. But they also had disadvantages like large high pressure boilers that could explode. They carried lots of highly explosive projectiles, propellant charges, depth charges, torpedoes and aviation gasoline that when exposed to damage could lead to secondary explosions. Modern merchant ships can be very hard to sink, or even stop, and a modern double hull tanker could be a particularly difficult.
SINK-EX RIMPAC 2000
To give some more recent data, using modern weapons, I’ll refer to RIMPAC 2000. One portion of the exercise included a missile firing exercise during which four decommissioned ships were sunk at the Pacific Missile Range Facility off the island of Kauai. The “Sink-Ex” operation involved firing of more than 100 missiles at the four target ships.
1.) ex-USS Worden (CG-18), a 7,800 ton full load (fl) guided missile cruiser commissioned in 1963. The former USS Worden sustained a continuous attack from two ships and from F-14 Tomcat and F-18 Hornet fighters, finally sinking 34 hours after the exercises started.
2.) ex-USS Buchanan (DDG-14), a 4,526 ton (fl) guided missile destroyer commissioned in 1962. Three Hellfire hits, three harpoon hits and a 2,400 pound laser-guided bomb hit were not enough to sink the ship, which required an additional 200 pounds of scuttling charges before sinking.
3.) ex-USS Ramsey (FFG-2), a 3,426-ton guided missile frigate commissioned in 1967. Missile and aircraft firing exercises involving nine ships and three different types of aircraft were required to sink ex-USS Ramsey and ex-USS Gen. Hugh J. Gaffey. Ex-Ramsey sank only after taking several surface and airborne harpoons.
4.) ex-USS Gen. Hugh J. Gaffey (AP-121), a 20,120-ton full load transport commissioned in 1944 that had once been the Coast Guard manned transport USS Admiral W. L. Capps (AP-121). Built to WWII merchant standards, Ex-Gaffey sunk nine hours after taking a total of 13 bombs.
While I can't claim it is dead accurate, I developed a rule of thumb when developing a rudimentary naval war game. It might serve as a useful metric until something better comes along: On average to have high confidence of sinking another ship, you need to put one pound of bombs or shells on target for every ton of ship. To disable them, it usually only takes about a tenth of that.
Torpedoes, considering only the warhead, are more effective. The weight of the warhead exploding at the side of the ship is about four times as effective as the same weight of ordinance, including explosive, bomb and shell casing, delivered above the waterline. Torpedoes exploded under the keel are about ten times as effective as the same weight delivered above the waterline.
Using this metric, sinking a 2000-ton WWII destroyer takes about 36 hits from a destroyer's 5-inch guns, about 18 hits from a light cruiser's 6-inch guns, 8 hits from a heavy cruisers 8-inch guns, or only one hit from a battle ship's 16-inch. A torpedo with a 500-pound warhead would usually sink a 2,000-ton ship (unless it hit at the extreme ends of the ship). That appears pretty close to actual experience.
Can the Coast Guard do it?
The Coast Guard's only current ships with any hope of stopping a medium to large ship are those armed with 76 and 57 mm guns. That in itself is not a good plan, because when the capability is needed, they are unlikely to be available. (I hope no one really expects to stop a determined attack by fast roping onto the deck.)
For the cutter faced with trying to stop a determined attack, I can only suggest that the CIWS may also be useful. I recommend targeting the rudder and the engine room near the waterline and maybe hitting the bow, too, in hopes of setting off any explosive there.
In terms of making the 76 and 57 mm more effective, the Coast Guard might consider ensuring the ships have available a very high velocity penetrating round, possibly using a discarding sabot as a means to get into a large diesel engine and destroy it. I know such a round is made for the 76mm, although it probably isn't in the USN system. (I don't know about the 57 mm.)
The real answer is to arm the more numerous and more readily available smaller cutters with a weapon that can reliably stop a ship of substantial size.
I think a modified Mk 46 could be an effective ship stopper, but if you absolutely, positively have to stop a ship immediately, nothing beats a heavy weight torpedo.
Consider the following case from the US Navy.
"On Monday, June 14, 1999, the Australian Collins class submarine HMAS Farncomb fired a Mark-48 war-shot torpedo at the 28-year-old former Destroyer Escort (actually a frigate) Torrens. The firing was part of the Collins class trials requirements and was designed to validate the submarine's combat system. The submerged Farncomb fired the Mark-48 torpedo at the stationary hulk of the 2700-ton Destroyer Escort from over the horizon. The plume of water and fragments shot some 150 meters skyward as the blast of the torpedo cut the ship in two. The stern section sank rapidly after the torpedo hit; the bow section remained afloat, but sank sometime later.
"The torpedo warhead contains explosive power equivalent to approximately 1200 pounds of TNT. This explosive power is maximized when the warhead detonates below the keel of the target ship, as opposed to striking it directly. When the detonation occurs below the keel, the resulting pressure wave of the explosion "lifts" the ship and can break its keel in the process. As the ship "settles" it is then seemingly hit by a second detonation as the explosion itself rips through the area of the blast. This combined effect often breaks smaller targets in half and can severely disable larger vessels.
"The Mark-48 torpedo used in this test is a variation of the MK-48 ADCAP (Advanced Capability) torpedo developed for the United States Navy."
In July 2012, a Mark 48 Torpedo was fired again, and it was successful again. According to the US Navy:
"The Royal Australian Navy's (RAN) Collins Class submarine HMAS Farncomb has successfully sunk a target ship, the 12,106-tonne former USNS (United States Navy Ship) Kilauea in Hawaii.
"Farncomb fired one Mark 48 Torpedo and achieved a hit just below the bridge of the ship as part of a sinking exercise, or "SINKEX," at Exercise Rim of the Pacific (RIMPAC) 2012.
"The former USNS Kilauea broke into two parts and sank about 40 minutes later."
 The primary source I used was the US Navy Report of War Damage series available here. The same index also includes reports of individual ship damage and reports of damage to British warships. I would also recommend the "Destroyer Report: Gunfire, Bomb and Kamikaze Damage, 17Oct41-15Aug45" which includes annotated damage control plates.