Wash the apples and orange thoroughly. Quarter the apples and slice the orange, but do not peel the orange.
In a large pot, combine the apples, orange slices, cinnamon sticks, whole cloves, and allspice. Add the water to the pot, ensuring that all the fruits are submerged. (You can add more water if needed.)
Bring the mixture to a boil over high heat. Once boiling, reduce the heat to low and let it simmer covered for about 2 hours.
After 2 hours, use a potato masher to mash the fruits to release their flavors. Let it simmer for an additional 1 hour.
Use a fine-mesh sieve or cheesecloth to strain the cider into a clean pot, removing all the solid pieces.
Return the strained liquid to heat, and add the sugar, stirring until completely dissolved. Taste the cider and add more sugar if desired, depending on your sweetness preference.
Serve the apple cider warm, garnished with cinnamon sticks or orange slices. Alternatively, allow it to cool and then refrigerate. Consume within 5-7 days for the best flavor. Can be served cold or reheated as desired.