VIENNA (Reuters) - The U.N. nuclear watchdog said on Thursday it had again failed to clinch a deal in talks with Iran this week on investigating suspected atom bomb research by the Islamic state.

The lack of a breakthrough in Wednesday's meeting in Tehran, though expected by Western diplomats, represented a new setback for international efforts to resolve a decade-old dispute over Iran's disputed nuclear program.

It comes before six world powers and Iran are due to meet for negotiations in Kazakhstan on February 26 over the Islamic state's atomic activities, which the West fears are aimed at developing nuclear bombs. Iran denies this.

Herman Nackaerts, deputy director general of the International Atomic Energy Agency, told reporters after he and his team returned to Vienna the IAEA was committed to continuing talks but needed more time to reflect on the way forward.

The two sides "could not finalize the document," he said, adding no date had yet been set for a next meeting.

(Reporting by Fredrik Dahl; Editing by Janet Lawrence)