Elendilmir was lost at the great massacre in the east, the Disaster of the Gladden Fields, when King Elendil's eldest son, Isildur, and the Dunedain of Arnor were marching home after the defeat of Sauron. The company were ambushed by a large group of Orcs from the Misty Mountains who slaughtered many including Isildur, with only three men to come out alive. It is a mystery exactly how Elendilmir was lost, but it is known that the One Ring was found at the bottom of the Gladden River where it had slipped off Isildur's finger when he was killed. This may be where Elendilmir was lost.
In the Unfinished Tales (later published by Christopher Tolkien) it is stated that Aragorn II finds the Elendilmir in the tower of Orthanc. After Sauron's defeat, Aragorn (with the help of Gimli the Dwarf) finds a hidden door and a steel cabinet. In that cabinet they find a small golden tube without any inscription (the One ring was most likely hidden in here by Isildur), and the Elendilmir. Aragorn brings it back to Minas Tirith and wears it once again, but only on special occasions as he will not risk any more harm to it.
Another version of the Gemstone was made at a later date by Isildur's descendants, a second Elendilmir, commemorating King Elendil and Isildur himself. It was borne roughly around TA 2940, or most likely later.