Narrated Ibn `Abbas: When Allah's Messenger (ﷺ) came to Mecca, he refused to enter the Ka`ba with idols in it. He ordered (idols to be taken out). So they were taken out. The people took out the pictures of Abraham and Ishmael holding Azlams in their hands. Allah's Messenger (ﷺ) said, "May Allah curse these people. By Allah, both Abraham and Ishmael never did the game of chance with Azlams." Then he entered the Ka`ba and said Takbir at its corners but did not offer the prayer in it.
Source: sunnah.com
Other translations
Arabic
حَدَّثَنَا أَبُو مَعْمَرٍ، حَدَّثَنَا عَبْدُ الْوَارِثِ، حَدَّثَنَا أَيُّوبُ، حَدَّثَنَا عِكْرِمَةُ، عَنِ ابْنِ عَبَّاسٍ ـ رضى الله عنهما ـ قَالَ إِنَّ رَسُولَ اللَّهِ صلى الله عليه وسلم لَمَّا قَدِمَ أَبَى أَنْ يَدْخُلَ الْبَيْتَ وَفِيهِ الآلِهَةُ فَأَمَرَ بِهَا فَأُخْرِجَتْ فَأَخْرَجُوا صُورَةَ إِبْرَاهِيمَ وَإِسْمَاعِيلَ فِي أَيْدِيهِمَا الأَزْلاَمُ فَقَالَ رَسُولُ اللَّهِ صلى الله عليه وسلم " قَاتَلَهُمُ اللَّهُ أَمَا وَاللَّهِ قَدْ عَلِمُوا أَنَّهُمَا لَمْ يَسْتَقْسِمَا بِهَا قَطُّ ". فَدَخَلَ الْبَيْتَ، فَكَبَّرَ فِي نَوَاحِيهِ، وَلَمْ يُصَلِّ فِيهِ.
sunnah.com