ELLEN and Bob Jaffe – who got fellow Palm Beachers to invest millions with Ponz scum Bernard Madoff – yesterday stepped down as co-chairs of the prestigious Discovery Ball at the Breakers Feb. 21. Nobody is surprised. Earlier this month at Mar-a-Lago, Jaffe was called a “dirty bastard” and nearly pummeled by Nine West founder Jerome Fisher, who lost $150 million. The upcoming fund-raiser for the Dana-Farber Cancer Institute of Boston is expected to draw a number of Madoff’s victims. Replacing the Jaffes are Howard and Michelle Kessler, who last year hosted a Hillary Clinton fund-raiser.