(Adds details, share rise, byline)

By Nadia Damouni and Greg Roumeliotis

NEW YORK, March 12 (Reuters) - Advent Software Inc, an accounting software company, is exploring a sale and has hired Frank Quattrone's investment bank Qatalyst Partners to lead the effort, several people familiar with the matter said on Tuesday.

Advent shares jumped on the news and were up 11 percent in late afternoon trading in New York, giving the company a market value of close to $1.5 billion.

Advent has attracted interest mostly from private equity firms, and the process is in the second round of bidding, the people said.

There is a substantial price gap between the company and some of the potential buyers and it remains unclear whether the auction will lead to a deal, said the sources, who asked not to be named because the process is not public.

Representatives for Advent Software and Qatalyst Partners did not immediately respond to requests for comment.

Founded in 1983, San Francisco-based Advent provides software and services to the global investment management industry. It posted revenues of $358.8 million in 2012, up 10 percent from 2011.

The company has embarked on a cost-cutting campaign to improve its profit margins in the face of slower revenue growth, as many of its customers, which include mutual funds, hedge funds and investment advisers, have been reluctant to spend following a long period of economic uncertainty.

Earlier this year, private equity firm TPG Capital LP agreed to buy a competitor of Advent, Eze Castle Software, from ConvergEx Group for an undisclosed amount. Asiff Hirji, a partner with TPG, has been a director on the board of Advent since 2011. (Reporting By Nadia Damouni and Greg Roumeliotis in New York, Editing by Soyoung Kim, Gerald E. McCormick and John Wallace)